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Overview 


This report documents the implementation and early impacts of Year Up—a national sectoral 
training program for urban young adults aged 18-24. Operated by an organization of the same 
name, Year Up provides six months of full-time training in the IT and financial service sectors 
followed by six-month internships at major firms. The full-time program provides extensive 
supports—including weekly stipends—and puts a heavy emphasis on the development of 
professional and technical skills. 


Year Up is one of nine programs in the federally sponsored Pathways for Advancing Careers and 
Education (PACE) evaluation. It is among the most intensive workforce training programs tested 
to date. More than half (59 percent) of the program’s $28,290 per-participant cost is funded by 
employer payments for interns. 


Using a random assignment design, the study found that Year Up increased receipt of 
employment and training services. Compared to control group members who were not able to 
access the program, treatment group members were more likely to report that their classes 
used active learning methods, taught life skills, and were relevant to their lives and careers. 
Most importantly, young adults with access to Year Up had higher average quarterly earnings in 
the sixth and seventh quarters after random assignment—the confirmatory outcome selected 
to gauge Year Up’s overall success for this report. Persisting over a three-year follow-up period, 
Year Up’s earnings impacts are the largest reported to date for workforce programs tested 
using a random assignment design. 


Primary Research Questions 
The Year Up evaluation addresses three main questions: 


e Was the Year Up program model implemented as designed across the program’s eight 
offices? 


e How did employment, training, and related service experiences differ between young 
adults randomly selected to participate in Year Up (the treatment group) and those who 
could not access the program (the control group)? 


e What were the effects of access to Year Up on earnings, career-track employment, 
postsecondary education, and related outcomes? How did these effects vary across 
subgroups and across the eight local offices? 


Purpose 


As opportunities for college graduates have grown in recent decades, prospects for young 
adults without postsecondary credentials have steadily worsened. With few chances at well- 
paying jobs, millions give up on school and withdraw from the labor force, often leading to 
long-term disconnection and life-long disadvantages. The societal costs of disconnection are 
high, including higher rates of crime, substance abuse, and intergenerational poverty; 
diminished tax revenues; and the loss to employers of millions of workers who otherwise could 
help to close skills gaps in growing industries. 
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The PACE evaluation of Year Up provides an opportunity to build on promising evidence for 
other well-implemented career pathways approaches with a strong sectoral focus. It assesses 
the possibility that more intensive and comprehensive interventions—addressing both general 
and occupation-specific skills, using work-based learning, and fully engaging employers—will 
prove more effective than lower-intensity programs studied in the past. 


Key Findings and Highlights 
Highlights from the implementation and early impact findings include: 


e Year Up implemented its program with high fidelity to its design. Site visits to four of 
Year Up’s eight local offices found all major program components implemented and 
functioning well. Quantitative indicators attest to strong implementation on outcomes 
such as recruitment, retention, internship slots, and post-program employment. 
Although generally high, outcomes varied somewhat across offices. 


e Year Up had positive effects on the scope and nature of services received by its 
students. More than half (57 percent) of control group members pursued training, 
mostly at community colleges. However, nearly all treatment group members (96 
percent) participated in Year Up, and they were far more likely to cite promising 
instructional approaches; take courses in life skills; accrue related work experience; and 
receive an array of support services. 


e Effects on earnings were consistently large and positive. This finding applies to the 
overall sample, to all participant subgroups analyzed, and to all eight Year Up local 
offices. Year Up increased average quarterly earnings by $1,895 (53 percent) in the sixth 
and seventh quarters after random assignment—the pre-specified confirmatory 
outcome for this analysis. Large positive effects persisted through the third follow-up 
year. Impacts also were positive for an array of other indicators of early career progress, 
although effects on college persistence were mixed. 


Methods 


The Year Up evaluation includes implementation and impact studies. The implementation study 
uses data from the program’s management information system; interviews with local staff, 
participants, and other stakeholders; an 18-month participant follow-up survey; and other 
sources. For the impact study, Year Up offices in all eight metropolitan areas recruited, 
screened, and randomly assigned a total of 2,544 young adults to treatment (1,669) and control 
(875) groups in 2013-2014. The evaluation team measured outcomes using data from 
employer-reported wage data maintained in the federal National Directory of New Hires, 
college enrollment records from the National Student Clearinghouse, and the 18-month follow- 
up survey. 


Prior to estimating Year Up’s impacts, the research team published an analysis plan specifying 
key hypotheses and outcome measures and registered those outcomes. An essential principle 
in the analysis plan was to organize and discipline the number of statistical tests conducted to 
avoid the problem of “multiple comparisons,” whereby a potentially large number of tests 
could reach conventional levels of statistical significance by chance, even if there were no effect 
on any outcome. 
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Executive Summary 


As opportunities for college graduates have grown in recent decades, prospects for young 
adults without postsecondary credentials have steadily worsened. With few chances at well- 
paying jobs, many give up on school and withdraw from the labor force. Short spells of 
withdrawal often lead to long-term disconnection and life-long disadvantages. The societal 
costs are high: lack of opportunity is associated with higher rates of crime, substance abuse, 
health problems, single parenting, and economic dependency. Tax revenues are diminished, 
and employers are deprived of millions of workers needed to close skills gaps in growing 
industries.’ 


This report provides encouraging evidence on the implementation and early impacts of Year 
Up—a national sectoral training program for urban young adults aged 18-24 with a high school 
diploma or equivalent. Operated by an organization of the same name, Year Up provides six 
months of full-time customized training in the IT and financial service sectors followed by six- 
month internships at major firms. The full-time program provides extensive supports— 
including weekly stipends—and puts a heavy emphasis on professional, as well as technical, 
skills. Employer payments to Year Up for interns financed 59 percent of the program’s $28,290 
per participant cost. The program grew rapidly following its inception in Boston in 2000, 
increasing both in the number and size of local offices. During the study period—2013-2014— 
Year Up served over 3,500 young adults in eight metropolitan areas. 


Year Up is one of nine programs Abt Associates is evaluating in the federally sponsored 
Pathways for Advancing Careers and Education (PACE) evaluation. Each of the nine evaluations 
includes implementation and impact studies. For the Year Up study, the evaluation team 
randomly assigned 2,544 young adults to treatment (1,669) and control (875) groups. This 
report covers the program’s implementation and early impacts, with follow-up centering on 1.5 
years and extending to 3 years for some outcomes. Future reports will examine program 
impacts for up to 6 years after program entry and include analyses of the program’s costs and 
benefits. 


Findings show that Year Up implemented its program with high fidelity to its design and 
substantially increased the training, support, and employment services young adults received. It 
had a large positive impact on the confirmatory outcome selected for this report—average 
quarterly earnings in the sixth and seventh quarters after random assignment. Average 
quarterly earnings were $1,895 higher for the treatment group ($5,454) than for the control 
group ($3,559)—a 53 percent impact. Impacts diminished but remained large (about 40 
percent) over the following year. As documented briefly in this summary and at more length in 
Chapter 1, these earnings impacts are the largest reported to date for workforce programs 
tested using a randomized controlled trial (RCT) design. 


See Chapter 1 for literature review and citations. 
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Program Overview 


Gerald Chertavian founded Year Up in 2000 to serve economically disadvantaged urban young 
adults ages 18 to 24.* The program targets young adults with a high school diploma or 
equivalent who are motivated and who, with assistance, can overcome challenges and 
successfully enter careers in fast-growing technical occupations. Year Up and other advocates 
have labeled this population “Opportunity Youth” to emphasize the positive contributions they 
could make, given effective interventions. 


Year Up’s original model—which it calls the “core program” — is a free-standing program 
operating in eight urban areas: Atlanta, Boston, Chicago, New York, Providence, the San 
Francisco-San Jose Bay Area, Seattle, and Washington DC. Abt Associates is evaluating the core 
program in all eight locations for PACE. ° 


The program teaches students high demand technical and professional skills, connects them 
with employers, and provides college credits via agreements with local college partners. A 
national team headquartered in Boston provides operations assistance (e.g., human resources, 
marketing, and accounting); facilitates cross-site knowledge building; and provides oversight 
and troubleshooting of local implementation. 


Payments from employers for interns cover 59 percent of Year Up’s costs. Nearly all of the 
balance (39 percent) of revenue comes from foundations and other private-sector donors, and 
only two percent is from public agencies. 


Recruiting from the general community, the program enrolls participants twice (in March and 
September) each year. During the first six months of the program—the “Learning and 
Development (L&D) Phase”—students attend 


courses at Year Up full-time. Training addresses Vear Upiprovides a very Consistent stucure = 


[It's not loose. You know what to expect: [the 


both occupation-specific and general skills. The hours are] 8:30-3:30, your stipend is this, and 
focus of technical training varies by office and [there is] a particular routine. Some of our young 
cohort. Fields include information technology (IT, people are coming from environments where there 
the most common emphasis), business operations was never a routine: you could show up at high 

! Bebe ; Pilastsynuet PerariOns: school or not show up, do whatever you want to 
financial operations, software development, and do. But Year Up creates a culture similar to the 
sales and customer support. General skills courses culture we are trying to have them assimilate in.” 
focus on professional and business communications —Year Up staff member 


skills. Students gain experience in writing, giving 

presentations, interacting with clients and colleagues, and developing critical thinking skills. 
Year Up sites work with local partner colleges so that students can earn college credit for their 
participation in Year Up. 


See Chertavian (2012) on Year Up’s founding and development. 


Since launching the PACE evaluation, Year Up also has launched pilots of an adaptation of the core program 
for college settings—the Professional Training Corps (PTC)—aimed at reducing costs and enhancing scalability. 
Abt Associates is collaborating with researchers at the University of Pennsylvania and Year Up on a separate 
evaluation of PTC. 
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Year Up’s “high support, high expectations” model provides extensive services and sets high 
standards for professional behavior. Each incoming cohort of 

young adults is organized into groups of about 40 students and “The main thing was that | could be 
staff called “learning communities” to foster supportive social part of something, not so alone. [It] 
connections. All participants receive advising from Year Up staff aeons I could be part 
members, and every staff member is expected to serve as a ny 

student advisor/coach in addition to other duties. Participants 
also receive mentoring from outside professionals working in 
related occupations. Each local office maintains a team of social workers who provide direct 
services and referrals to help students address varied life challenges. Students receive weekly 
stipends (typically $150 during the first half and $220 during the second half of the program) to 
help cover transportation and other program-related expenses. Students sign a formal contract 
specifying standards for professional behavior. Infractions trigger stipend reductions and can 
lead to dismissal from the program. The program teaches techniques for constructively giving 
and receiving feedback. It hones these skills in “Friday feedback” sessions and encourages 
practice throughout the week. 


—Year Up participant 


In the second half of the year—the “Internship 
Phase” —students intern at local firms, often Fortune 
500 companies. Students work at their internship sites 


“[My internship] is exciting— like it. I’ve 
learned a lot. ... | hardly knew about 
SharePoint. Right now, | could say I’m the go- 


full-time for four-and-a-half days a week. Students to gal. My manager, she’s pretty impressed 
return to Year Up each week for a half-day skills that | caught on so quickly. I've been practicing 
workshop during which they share their internship CUA Ne ETN E EIE LS GUE TS 


. . something, | have to figure it out. I'll just 
experiences and plan for education and careers after Google it and do it.” 


graduation from the program. Towards the end of 

internships, the emphasis on job search and 

placement intensifies. Active efforts to support job 

search and placement continue for up to four months after graduation. 


—Year Up participant 


Evaluation Design 


To implement the impact study’s random assignment design, staff in Year Up’s eight* core 
offices recruited, screened, and randomly assigned a total of 2,544 young adults to treatment 
(1,669) and control (875) groups at a 2:1 ratio in 2013-2014. The design generated a treatment 
group sufficient to fill all seats in each office over two recruiting cycles. 


Year Up staff actively engaged treatment group members to encourage follow-through on 
enrollment, and virtually all (96 percent) did enroll. Control group members were subject to a 
three-year embargo on Year Up participation but could receive other training and supports in 
the community. An analysis of baseline data shows that the treatment and control groups were 
statistically indistinguishable on an array of demographic, economic, and psycho-social 
characteristics (e.g., motivation, self-confidence, and social support). The career pathways 


3 Although Year Up has two offices in the Northern California Bay Area—in San Francisco and San Jose—key 


operations are jointly administered, and the program typically groups the two offices as a single location in 
performance measures. 
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framework developed for PACE (Fein 2012) and Year Up’s theory of change guided the 
evaluation team in selecting key research questions and outcome variables. 


The impact study draws on three data sources: (1) employer-reported Unemployment 
Insurance wage data maintained in the federal National Directory of New Hires (NDNH), 

(2) college enrollment records from the National Student Clearinghouse (NSC), and (3) an 
18-month follow-up survey. Survey response rates were 78 and 73 percent for treatment and 
control group members, respectively. 


For the implementation study, the PACE team conducted interviews and observations at four 
local Year Up offices and at the program’s national office in Boston. Fieldwork also included in- 
depth interviews with 19 treatment and 13 control group members and with employer and 
college partners. Quantitative analyses of implementation draw on extracts from Year Up’s 
management information system, a survey of local Year Up staff, and the 18-month follow-up 
survey. 


Implementation Findings 


Results from field research and quantitative analyses of program data show that the eight local 
Year Up offices fully implemented all program components and generated strong results on a 
series of implementation indicators. This is a substantial achievement given the number and 
complexity of program elements, varying local environments, and the need for tight 
coordination across many services. 


e Year Up achieved high levels on measures of implementation for recruitment, retention, 
internships, revenue, and post-program employment. It earned high praise from 
employers. 


All offices met the study’s requirement for expanding recruitment by 50 percent for PACE. They 
did so while maintaining applicant quality, admitting only one in six applicants as in the past. 
Nearly all (96 percent) treatment group members actually enrolled in training (Exhibit ES-1). 
Retention was high: 75 percent of the treatment group (78 percent of those enrolling) 
completed the program. Staff diligently enforced Year Up’s contract: 96 percent of enrollees 
received at least one infraction, and 45 percent received 10 or more infractions. 


The program placed 99 percent of L&D completers in 


internships and generated $22,404 in revenue from “I like [the contract]. | think it holds every 
student accountable. If they didn’t have it, 


em ployers for the average intern. Averaged across all Bier nad betas (mnerins mental 
participants (including those dropping out before don't think that would be fair. Now that 
internships), this revenue financed 59 percent of the we're on internship, we're dressed 
program’s $28,290 per participant cost. Foundations and professional because of Year Up ... not 


using slang, getting to work on time, 


private donors provided the vast majority of the balance, actually turning our work in on time too.” 


and only two percent of operating funds came from 


public agencies. —Year Up participant 


Four months after graduation, 83 percent of graduates were employed. Of those working, 

89 percent were employed full-time, 88 percent were in an occupation relevant to their Year 
Up’s training, 41 percent had jobs with their internship sponsors, and 77 percent were earning 
$15 per hour or more. 
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Exhibit ES-1: Progress through Year Up by Treatment Group Members 


ASSIGNMENT TO YEAR UP: 
100% 


DID NOT ENROLL IN ENROLLED IN PROGRAM: 
PROGRAM: 96% 


4% 


COMPLETED LEARNING & 
DEVELOPMENT PHASE: 


81% (85% OF ENROLLEES) 


COMPLETED INTERNSHIP & 
GRADUATED: 
75% (78% OF ENROLLEES) 


EMPLOYED AT 4-MONTH 
YEAR UP FOLLOW-UP: 
62% (83% OF GRADUATES) 


Abt Associates 


Percent of enrollees who 
received contract penalty 


1+ infraction: 96% 
10+ infractions: 45% 


Percent of employed alumni 
working 


Full-time: 89% 

In targeted occupation: 88% 
At internship sponsors: 41% 
For $15+/hour: 77% 
(Average wage: $17.41/hour) 
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Employers’ perceptions of Year 

Up interns were highly bart a they = all ie Heh ee cena ae ae th 
professionalism. ... Year Up interns hold their own and get hired over the 
favorable. Seve ral yet ploye is stereotypical college graduate from a stable home. Those [college] kids don’t 
noted that experience with know how to interact with a manager. They don’t know how to make small 
Year Up interns had led themto _ talkin an elevator or with a director. They don’t know how to write a 

revamp career pathways in professional email that doesn’t use text abbreviations, but the Year Up interns 


: . . do.” 
skilled technical occupations to 
: “The Year Up kids ran circles around the 45-year-old professionals. They did 
create new career entry points 
ad : everything we asked them to. They did it with a smile. They did it confidently.” 
at lower levels. One large firm 


heavily re-oriented its entry- 
level hiring practices to recruit 
sub-baccalaureate-level 
interns. For another, creating 
assignments for interns stimulated the realization that a substantial layer of tasks currently 
handled by mid-level employees could be performed by Year Up graduates. 


“They're eager and excited. They want to learn and engage. It helps with our 
employees as well, keeping up the morale.” 


—Internship employers 


e Treatment group members reported experiencing promising instructional practices more 
often at Year Up than control group members experienced at other training providers. 


Virtually all treatment group members participated in Year Up, and over half of control group 
members received education and training from other providers—mostly community colleges. In 
the 18-month survey, treatment group members were more likely than control group members 
who pursued training elsewhere to say that the instruction they received included practices 
seen as promising in the workforce field. As shown in Exhibit ES-2, Year Up participants were 
more likely to say that their classes mostly involved 
project- and group-based work and less likely to say that “There's more interaction than lecture. You 
classes mostly involved lectures. The former also were can talk more, be more communicative ... 
more likely than the latter to indicate that their classes is SESS Ueto) neu Sas 

. . ; : [also liked] that it was more hands on. 
stressed active learning methods (e.g., discussion and 
projects), and that classes were relevant to important life 
pursuits. More students in the treatment group reported 
taking courses specifically focused on life skills than did students in the control group. 


—Year Up participant 


More treatment than control group students said they received supports while in education 
and training. Year Up students were nine percentage points more likely than control group 
students to receive academic advising, 18 percentage points more likely to receive tutoring, and 
nine percentage points more likely to receive financial aid advising (not shown in exhibit). Year 
Up participants were 13 percentage points more likely to cite receipt of grants or scholarships 
and 22 percentage points less likely to take out loans than control group members at other 
schools. 


e Year Up increased receipt of key career supports. 


The 18-month follow-up survey also measured receipt of selected services from any source for 
all sample members, including those not enrolling in school. The bottom panel in Exhibit ES-2 
summarizes these impacts. Year Up increased the likelihood of receiving career counseling by 
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39 percentage points; supports for meeting school, work, or family responsibilities by 33 points; 
and job search and placement assistance by 43 points. 


Exhibit ES-2: Education and Training Experiences and Receipt of Supports after Random 
Assignment 


Among respondents receiving education 
and training, 


Most/all class time spent: 
In group projects & 


V/1111'* 


30 
Wi Treatment Group 


Control Group 


43 
In group work 


LLLLLLL11112 * 


34 


VLLLLLLLLLLLL LL LA +2 


Listening to 
lectures 


Classes often use active 
learning methods 


56 
VLLLLL11112) ® 


Strongly agree classes 
relevant to life/career 


69 
LLLLLLLLLLLLLLLLLLL2* 


Took 1+ class in life 76 


Bs VILLILILLLLL 1) * 


Among all respondents, 
ever received: 


59 
Career counseling 
Help arranging 
supports to meet 
school, work, or 
family responsibilities 
61 


Job search/ 
placement assistance 


60 70 80 


Percent of Respondents 
SOURCE: PACE 18-month follow-up survey. 
NOTES: All treatment-control differences are statistically significant at the 1-percent level (two-tailed tests). Statistics in the top 
panel apply to the subset of survey respondents attending education and training since random assignment (1,035 treatment, 367 
control) and are not regression-adjusted. Statistics in the bottom panel apply to the full survey sample (1,301 treatment, 638 control) 
and are regression-adjusted. 


Abt Associates Executive Summary 1 pg. ES-vii 


Year Up 
Implementation and Early Impact Report 
PACE 


e Year Up’s national team used varied strategies to maintain fidelity to the program model 
following a period of rapid growth. 


The PACE years (2013-2014) followed a period of rapid growth in the number and size of local 
Year Up offices (see Fein 2016a). This growth brought increased variation in local approaches to 
key program strategies. It was not always easy for program leaders to distinguish positive 
adaptations to local conditions from less productive departures from the program model. Rapid 
growth also heightened the challenge of knitting together increasingly specialized staff 
members. 


One strategy Year Up’s leaders used to maintain model fidelity was 
promoting alignment on the program’s mission—closing the 
“Opportunity Divide” for urban young adults. They fostered such e Respect and value others 
alignment through continuous engagement with staff, participants, e Build trust, be honest 
and other stakeholders (e.g., employers, college partners, and : Te ae eg 
funders) in discussions of the program’s core values (see box). e Be accountable 


e Strive to learn 
e Work hard and have fun 


Year Up’s Core Values 


Fieldwork conducted during the PACE study found many instances 
of staff and students consciously trying to connect these values to 
their daily activities. For example, “strive to learn” is one core Year 
Up value. Intentional efforts to apply this value led students to take responsibility for their 
studies, local staff to put energy into improving services, national staff to create program-wide 
mechanisms for continuous improvement, and all to give and accept constructive feedback. 
Another salient facet of Year Up’s culture is the 


. . “One thing that | really, really like about Year U 
unusual degree to which business values, language, : ee c 


is the way they do feedback. ... | have never had 


and tools are incorporated throughout the program. feedback that way. They focus on your strengths 
F and commend you for what you did right and your 
Leaders also used the program’s performance strong points. Then they say, ‘With that in mind, | 
measurement system—FM-RADIO—to promote would encourage you to do this.’ See? So it’s not 
alignment on common goals across different offices saying, ‘But you do this all wrong.’ They say it in a 


way that’s going to encourage somebody to keep 


7 5 < , 
and a diverse staff. ° This tool provided a way to that growth area in mind, | like that a lot.” 


monitor and troubleshoot varying approaches to 
services while encouraging innovative adaptation to 
local conditions. Its measures of multiple outcomes 
also helped to promote awareness of the relationships between work performed by different 
staff specialties—highlighting, for example, the connections between admissions screening, 
L&D retention, and internship revenue. 


—Year Up participant 


Another integrative element of Year Up’s culture is its deliberate cultivation of a strong 
business ethos. Fieldwork for PACE found pervasive evidence of such an ethos—as exemplified 


The FM-RADIO dashboard tracks outcomes for local offices on key indicators for: Financial Management 
(revenue covers costs), Retention, Admissions (recruitment), Development (fundraising), Internship sales, and 
positive Outcomes. During PACE, for example, local offices had to meet a 75-percent overall retention 
standard, secure internships for all L&D completers, and ensure that 85 percent of graduates were working 
full-time in Year Up target occupations for at least $15/hour and/or enrolled in post-secondary education at 
the four-month post-graduation time point. 
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in the energetic, entrepreneurial approach of its staff and the infusion of business language, 
concepts and tools throughout daily activities. Interviews with students, staff, and employers 
suggested that this aspect of the culture helps to motivate students to work hard and connect 
well with Year Up’s employer partners. 


Finally, national leaders took steps to promote consistency by standardizing some services and 
shifting authority to the national level. The 2013-2014 period brought initiatives to standardize 
key admissions processes, centralize curriculum planning, and foster a “customer solutions” 
approach to corporate relations aimed at expanding the number of interns large employers 
were willing to host. 


Year Up’s Early Impacts 


This section summarizes Year Up’s early impacts on earnings, employment, and postsecondary 
education. It also discusses findings on impacts for subgroups and local Year Up offices. The 
analysis estimates impacts for each outcome by calculating the difference between average 
values in the treatment and control groups. 


e Year Up generated a large, statistically significant increase in average earnings in the 
sixth and seventh quarters following random assignment (the confirmatory outcome). 


The study’s confirmatory hypothesis was that Year Up would increase quarterly earnings 
averaged across the sixth and seventh quarters after random assignment. The evaluation team 
chose this period to confirm short-term success because these quarters immediate follow Year 
Up’s four-month post-graduation employment services. Measures of average earnings include 
sample members who did not work (and thus had zero earnings) during each quarter. 


Year Up produced a statistically significant $1,895 (53 percent) positive impact on average 
quarterly earnings, which were $5,454 and $3,559 for treatment and control group members, 
respectively. This estimated impact, based on employment records in the NDNH, is very close to 
independent estimates from the 18-month follow-up survey. 


Exhibit ES-3 shows impacts for individual calendar quarters extending for three years after 
random assignment. It shows that large positive impacts persisted through the end of the third 
year. Overall impacts are $5,181 in Year 2 (a 39-percent impact) and $7,011 in the Year 3 (a 40- 
percent impact). By comparison, impacts for the most effective workforce programs rigorously 
tested to date have been smaller.° 


For example, the Sectoral Employment Impact Study (SEIS) reported some of the largest earnings impacts to 
date—$4,001 (29 percent)—for a pooled sample of adults in three sectoral programs in the second follow-up 
year (Maguire et al. 2010). The SEIS estimate for 18-24-year olds was $3,092. Among the three programs 
tested, Per Scholas, an IT-focused program with some similarity to Year Up, generated the largest impact for 
adults of all ages—S$4,663 (32 percent)—but had only a small ($1,339) and statistically insignificant impact for 
young adults. A recent replication also found substantial, if somewhat smaller, overall Year 2 impacts for Per 
Scholas but did not provide separate estimates for young adults (Hendra et al. 2016, Schaberg 2017). Although 
Project QUEST, a health care-focused sectoral program, increased earnings by $5,080 (22 percent) in Year Six 
for a general sample of adults, impacts for young adults were statistically insignificant and negative in sign 
(Elliot and Roder 2017). 
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Exhibit ES-3: Impact on Average Earnings in Successive Follow-up Quarters 
Average Quarterly Earnings ($) 
7000 


+1.739 +1,788 


+1,781 


wee 


+1,783 +1,703 


wee 


+2,006 


hk 


6000 


wee 


Impacts ——> +1,794 


kik 


5000 
4000 


3000 
@ Treatment Group 


2000 A Control Group 


1000 In-Program Quarters 


si “1,362 1,674 -2,002 


Q2 Q1 Q0 Qi @Q2 Q3 Q4 @5 QE QZ QB QI Qin Qi1i1 
Follow-up Quarter 


SOURCE: Match to wage records in the National Directory of New Hires for 1,638 treatment and 858 control group members. 
NOTES: Impacts appear as numbers giving differences in average earnings for treatment and control group members in each 
follow-up quarter. *** Impact in a two-tailed test is statistically significant at the 1-percent level, ** at the 5-percent level, * at the 10- 
percent level. 

Training programs typically produce negative earnings impacts in the first follow-up year (the 
program year), as participants prioritize training over work. Year Up was no exception, with a 
$5,338 reduction in the first follow-up year. The Year 1 loss was more than offset by positive 
impacts totaling $12,192 over the next two years, however. Moreover, in Year 1 treatment 
group members also received an average of $7,172 in Year Up stipend payments not included in 
the estimated earnings impacts. 


e Survey data indicate that Year Up’s impacts on earnings reflect increases in average 
hourly wages and hours worked rather than increases in the percent employed. 


Although employment rates were identical (at 74 percent) for the control and treatment groups 
in the 18-month follow-up survey, Year Up increased the total hours worked by an average of 
3-4 hours per week relative to control group members (who worked about 24 hours per week 
on average). Employed treatment group members earned nearly $4 per hour more than 
employed control group members and were substantially more likely to be working in 
information technology (37 versus four percent) and in business and financial occupations 

(23 versus 14 percent).’ 


Higher earnings contributed to improved financial circumstances: Year Up reduced the percent 
of individuals citing financial hardships in the past 12 months from 43 percent to 35 percent. 


e Impacts on postsecondary education were mixed. 


Provisions to co-enroll Year Up participants at local colleges led to a doubling in the proportion 
enrolled in college during the first follow-up year (see Exhibit ES-4). As Year Up graduates went 
to work in the second year, college enrollment in the treatment group fell to below that of the 
control group. 


7 Not shown in exhibit. 
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Exhibit ES-4: Impacts on College Enrollment and Postsecondary Credentials over the First Two 
Follow-up Years 


Percent of Sample Members 
80 


i Treatment Group Control Group 


18*** 


Quarters 0-3 Quarters 4-7 Any college Professional 
credential certification/license 
Any College Enrollment Received Post-Secondary Credential 


SOURCE: Match to college enrollment records in the National Student Clearinghouse for 1,668 treatment and 871 control group 
members. Credential receipt as reported in the PACE 18-month follow-up survey for 1,301 treatment and 638 control group 
members. 

NOTES: *** Statistically significant in a two-tailed test at the 1-percent level. 


Year Up produced a small increase in the average number of credits earned (4.0 credits) in the 
first 18 months after entering the program (not shown in exhibit). There was no impact on 
receipt of college credentials: only three percent of individuals in either group reported 
receiving a college credential since random assignment. On the other hand, Year Up increased 
receipt of industry certifications and licenses by 18 percentage points. 


e Year Up increased earnings for every subgroup of participants examined and for all eight 
local Year Up offices. The size of impacts nonetheless varied for a number of 
characteristics. 


Impacts on quarterly earnings in Quarters 6—7 were at least $1,000 and statistically significant 
in every subgroup across 11 characteristics examined. The size of effects varied substantially, 
however. Impacts were lower for participants who reported weaker high school performance 
and those who at study intake expected to work full time over the next few months. Such 
results suggest possible benefits from strengthened academic supports and help minimizing 
challenges arising from outside work during the program. Smaller earnings gains for African 
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Americans may signal that the well-documented disadvantages facing this group— including 
greater discrimination and social and economic challenges—also require greater effort to 
overcome (National Research Council 2014). 


Although motivation would seem to be critical to succeeding in Year Up, impacts were more 
favorable for participants who rated /ower on training commitment than for those who rated in 
the highest category on a 10-item commitment scale.° It suggests that the most motivated 
recruits were in a somewhat better position to succeed without Year Up—though they still did 
benefit from the program. The finding also raises the possibility that Year Up may be effective 
for less motivated young adults than the program historically has targeted. It hints that low 
motivation may be malleable for some young adults, given sufficient time in a highly structured 
and supportive program. 


Positive impacts in all eight local offices attest to Year Up’s replicability in different urban 
environments. That said, the size of impacts varied substantially across offices. This variation 
might arise from differences in local environments, the quality of implementation, or both. 
Considerable differences in average control group earnings across offices indicate that 
contextual factors—such as local population characteristics, labor markets, and access to 
transportation—could play an important role. But indications also suggest variability in 
implementation.” With only eight offices and many dimensions to context and implementation, 
it is not possible to identify contributing factors with confidence. 


Implications 


This report’s findings add to growing evidence on the promise of well-implemented career 
pathways approaches with a strong sectoral focus. These approaches tend to be intensive and 
comprehensive, address psycho-social skills as well as occupation-specific competencies, 
provide opportunities for work-based learning, engage employers, incorporate evidence-based 
practices, and emphasize continuous improvement. 


The PACE findings should allay concerns that programs like Year Up that screen for ability, 
motivation, and manageable life challenges necessarily are serving young adults who would do 
just as well without such interventions. Baseline statistics show substantial socioeconomic 
disadvantages in the young adults Year Up recruited, and the study’s random assignment 
design establishes that they truly benefitted from the program’s services. 


The PACE Training Commitment scale is based on ACT Inc.’s proprietary Commitment to College scale, 
modified in consultation with ACT staff to apply to training more broadly. The scale captures commitment to 
persisting in and completing training. Illustrative items, rated on a 7-point scale ranging from strongly disagree 
to strongly agree, include: “| am committed to finishing whatever educational program | am in” and “I would 
stop attending school if | found something more interesting to do.” 


For example, program completion rates ranged from 64 percent in one office to 85 percent in two offices, with 
the remainder in the 75-79 percent range. While for six offices 30-39 percent of interns were hired by their 
internship sponsors, outliers included one office with a 21 percent conversion rate and another with a 52 
percent rate. Contract enforcement was especially variable, with the percent cited for 10 or more fractions 
ranging from 21 percent to 60 percent. 
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Stringent screening does raise the important question of 


whether Year Up is appropriate for more than a narrow 
segment of Opportunity Youth. This study’s findings of 
large positive impacts for participants with weaker 
academic backgrounds, lower motivation, lower incomes, 
and a wide variety of demographic characteristics hint 


“| could say that I've really grown ... that’s 
what | mean by it changed my life. A year 
ago | wasn't thinking about nothing like this 
or being in contact with the people I’m in 
contact with or doing the things I’m doing. | 
feel like it's changing how | look at people 
now, how | look at situations.” 


that the program might be effective with lower-skilled, 


more disadvantaged populations. But they do not prove 


—Year Up participant 


that it would be more widely effective or establish how far beyond the current population 
wider benefits might extend. Future experiments testing the model with expanded target 


populations would be valuable. 


Year Up’s success in engaging employers 
demonstrates the potential for not only 
expanding opportunities for work-based 
learning in fast-growing professional 
occupations, but also for mobilizing private- 
sector financing of organizations that serve as 
intermediaries between newly skilled job 
seekers and employers. Thus, while 
exemplifying more familiar forms of 
engagement such as appointing business 
representatives to advisory boards and 
working together in identifying skills in 
demand and designing training programs, 
Year Up’s experience also highlights the 
potential for employers to play central roles 
also in funding and service delivery in 


“There are so many elements to the program that | feel 
honestly that if you were to remove one piece you'd have 
a different ... it’s like when you made the best strawberry 
upside down cake, and then you take a little bit of sugar 
out, and you're like, ‘Damn, this doesn’t taste the same.’ 
So the mission-driven staff is one main piece. The 
programming pieces are great, but you need real people 
that care enough to do these programming pieces well, 
and | think it’s really the staff that it boils down to. And you 
really, essentially need the internships. ... It's really that 
end piece, the internship experience, the job, that says, 
‘Okay, this is what you would've done.’ And having 
internship partners that believe in the model. ... 
[Employers] have to believe that we can provide them 
fantastic interns that can do the job, and we have to 
provide them fantastic interns that can do the job.” 


—Year Up staff member 


workforce programs for economically disadvantaged adults. 


As noted earlier, several employers noted that experience with Year Up interns led them to 
revamp career pathways in technical occupations to create new career entry points at lower 
skill levels. Although anecdotal, such reports show how employers can become comfortable 
substituting a trusted workforce intermediary’s “brand” for academic credentials in hiring. 


Looking Ahead 


While answering one set of questions about Year Up’s implementation and early impacts, this 
report raises many questions for future research. Future data collection and analysis for the 
PACE evaluation will address some of these questions, whereas others will require separate 


studies. 


Abt Associates 
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One series of questions for the ongoing PACE evaluation concerns how Year Up’s impacts will 
evolve over the longer-term: 


e Do impacts on earnings persist or fade? 


e To what extent do the program’s positive initial impacts on employment in entry-level 
career-track jobs position young adults to continue along career pathways in 
occupations Year Up targets? 


e What role does college play in these pathways, compared to young adults who did not 
go to Year Up? At what rates do Year Up graduates return to school and earn college 
credentials in the longer-term? 


e To what degree does career progress affect well-being in other domains? 


Future reports on program impacts covering up to six years of follow-up will address these and 
other questions. 


Another important set of questions concerns Year Up’s costs and benefits. A planned cost- 
benefit study will estimate the net financial returns from the perspectives of participants, 
government, the balance of society, and society as a whole (i.e., summing the first three). 
Positive earnings impacts and unique features of Year Up’s financial model raise interesting 
questions about longer-term costs and benefits. Although Year Up’s costs per participant are 
relatively high, revenue from employers covers a substantial share of costs, and dependence on 
public funds is minimal. From participants’ perspective, foregone earnings during the program 
year are more than offset by program stipends. The potential societal benefits are considerable: 
one set of simulations for the general U.S. population shows economic returns from averting 
disconnection of around $600,000 per youth (Belfield et al. 2012). 


In describing plans for Year Up, leaders emphasized that substantially expanding the program is 
the main priority going forward. Having demonstrated the effectiveness of the core program 
model, the organization has launched several initiatives to develop and test more scalable 
approaches (Fein 2016a). 


As part of these efforts, Year Up is exploring how an increased emphasis on college completion 
might help to ensure continuing career advancement after participants finish the program. The 
present study’s finding of negative impacts on college enrollment after graduating Year Up 
attests to the difficulty of combining full-time work and school. A newer Year Up model—the 
Professional Training Corps, which leverages college facilities and instructional capacity— 
provides a fertile field for exploring the tradeoffs. It creates intriguing possibilities for moving 
from two-partner sectoral models focused on career entry to three-partner models that add 
colleges to support further movement in career pathways. 
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As opportunities for young adults with college degrees have grown in recent decades, prospects 
for those without postsecondary credentials have steadily worsened.” With few chances at 
well-paying jobs, millions give up on school and withdraw from the labor force. Short spells of 
withdrawal often lead to long-term disconnection and life-long disadvantages. The societal 
costs of disconnection are high: lack of opportunity is associated with higher rates of crime, 
substance abuse, economic dependency, child abuse and neglect, and intergenerational 
poverty. Tax revenues are diminished, and employers are deprived of millions of workers 
needed to close skills gaps in growing industries. 


A wide range of interventions have aimed to improve workforce outcomes for disadvantaged 
youth and adults. Randomized controlled trials (RCTs) typically have found at most small 
effects, and positive findings have been difficult to replicate. Short-term training for entry-level 
jobs in growing occupational sectors has produced some of the best results to date. These 
sectoral programs generally target low-income adults who have high school diplomas and 
screen for motivation and other factors related to ability to benefit. 


This report provides evidence on the implementation and early impacts of Year Up, a sectoral 
training program for urban young adults ages 18-24 with a high school diploma or equivalent 
(Chertavian 2012). Operated by an organization of the same name, Year Up provides six months 
of full-time training in the information technology (IT) and financial service sectors followed by 
six-month internships at major companies. The program also provides extensive supports— 
including weekly stipends—and puts a heavy emphasis on professional, as well as technical, 
skills. Employer payments for interns finance 59 percent of the program’s $28,290 per 
participant cost. Year Up grew rapidly following its inception in Boston in 2000, increasing both 
in the number and size of local offices. In 2013-2014, the period that PACE cohorts participated 
in Year Up, the program served more than 3,500 young adults in eight urban areas around the 
country. 


Abt Associates is evaluating Year Up as part of the Pathways for Advancing Careers and 
Education (PACE) evaluation. ** The evaluation includes both an implementation study to 
examine Year Up’s design and operation and an impact study that uses a RCT design to 
estimate effects on earnings, employment, postsecondary training, and other outcomes. This 
report evaluates Year Up’s implementation and impacts over a follow-up period of up to three 
years after random assignment. Future reports will extend follow-up for up to six years. 


This chapter describes the PACE evaluation, reviews the research backdrop for Year Up, and 
outlines the report’s remaining chapters. 


0 The body of this chapter provides citations for this and the following paragraph. 


** For more information on the PACE study, go to www.acf.hhs.gov/opre/research/project/pathways-for- 


advancing-careers-and-education. 
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1.1. Pathways for Advancing Careers and Education (PACE) Evaluation 


Funded by the Administration for Children and Families (ACF) within the U.S. Department of 
Health and Human Services, the PACE evaluation is a study of nine programs that include key 
features of a “career pathways framework.” Initiated in 2007, PACE represents the first large- 


scale, multi-site experimental evaluation of career pathways programs. 


The career pathways framework guides the 
development and operation of programs 
that aim to improve the occupational skills 
of low-income individuals—primarily 
nontraditional students—by increasing their 
entry into, persistence in, and completion of 
postsecondary training. Central to 
accomplishing these improved outcomes, 
the framework articulates signature 
strategies for overcoming the barriers that 
nontraditional, occupational students often 
face. For example, key features of programs 
within this career pathways framework 
include having a series of well-defined 
training steps, engaging instruction, 
supportive services, and connections to 
employment (Fein 2012). 


Programs consistent with the career 
pathways framework typically have multiple 


Programs in PACE 


Bridge to Employment in the Health Care Industry at San 
Diego Workforce Partnership, San Diego, CA 

Carreras en Salud at Instituto del Progreso Latino, 
Chicago, IL 

Health Careers for All at Workforce Development Council 
of Seattle-King County, Seattle, WA 

Pathways to Healthcare at Pima Community College, 
Tucson, AZ 

Patient Care Pathways Program at Madison College, 
Madison, WI 

Valley Initiative for Development and Advancement 
(VIDA), Lower Rio Grande Valley, TX 

Washington Integrated Basic Education and Skills 
Training (I-BEST) program at three colleges (Bellingham 
Technical College, Whatcom Community College and 
Everett Community College), Washington State 
Workforce Training Academy Connect at Des Moines 
Area Community College, Des Moines, IA 

Year Up (Atlanta, Bay Area, Boston, Chicago, National 
Capital Region, New York City, Providence, Seattle) 


components. The multi-component nature of such programs reflects the reality that 
nontraditional students often face many barriers and need additional support to substantially 
improve their employment or other prospects. The career pathways framework is flexible, 
however, and not a specific program model. Thus, which components a local program adopts 


and how it implements them can vary greatly. 


Reflecting this diversity, each of the nine programs in the PACE evaluation represents a 
different program model. All share some program components that are part of the career 
pathways framework, but each also has distinct and unique elements, reflecting the target 
populations, occupational trainings offered, and industries of focus. Because of this variation, 
PACE evaluates and reports findings for each evaluated program individually.‘ 


The central goal of the PACE evaluation is to determine the effectiveness of each of the nine 
programs using a common evaluation design and conceptual framework. The most critical 
element of the evaluation design is random assignment of eligible applicants either to a 
treatment group that can access the career pathways program or to a control group that 


% All PACE-related reports can be found on www.acf.hhs.gov/opre/research/project/pathways-for-advancing- 


careers-and-education and www.career-pathways.org and www.career-pathways.org. 
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cannot. Random assignment ensures that the study’s treatment and control groups will be 
equivalent in their observed and unobserved characteristics, and that any systematic 
differences in their subsequent outcomes (i.e., the program’s impacts) can be attributed to the 
treatment group having access to program services. Systematic differences in outcomes due to 
the characteristics of individual members in each group can be ruled out. 


Consistent with this career pathways framework and the theory of change guiding the PACE 
Year Up evaluation, as described in Chapter 3, impact analyses concentrate on employment and 
training outcomes, although the study also analyzes effects in other domains. 


1.2 Challenges and Opportunities Facing Economically Disadvantaged Youth 


Before examining Year Up’s approach in detail (in Chapter 2), the remainder of this chapter 
describes the wider research backdrop to the present study. This section looks at statistics on 
economically disadvantaged youth and on the availability of jobs in the sectors programs like 
Year Up target. 


Target Population. Year Up and other workforce organizations have framed the case for 
investing in young adults in terms of a need to close the “Opportunity Divide” for “Opportunity 
Youth.” The concept of Opportunity Youth commonly refers to young adults who are 
disconnected from school and work. On the other side of the divide is unmet demand for 
workers with good soft skills and entry-level technical skills in fast-growing occupations. 


One widely-cited figure for Opportunity Youth is Belfield et a/.’s (2012) estimate of 6.7 million 
16- to 24-year-olds (17.3 percent of this age group) who spent little or no time in school or 
work in the prior year. Of this number, Belfield et al. estimate that about half (3.3 million) had 
some periods of school and work between ages 16 and 24. They suggest that this “under- 
connected” half of the population offers better prospects for education and training-focused 
interventions than do the “chronically disconnected” half. 


A growing body of evidence attests to the economic disadvantages facing youth lacking post- 
secondary credentials. One recent analysis estimated that, compared with young adults with no 
more than a high school credential, college graduates’ earnings were 34 percent higher in 1980, 
57 percent higher in 1990, 71 percent higher in 2000, and 78 percent higher in 2015 (Valletta 
2016). Among 18- to 24-year-olds in 2013 with only a high school credential, some 30 percent 
were neither working nor in school, compared to 13 and 9 percent, respectively, for those with 
an associate’s or bachelor’s degree (National Research Council 2014). Young adults without a 
college degree also experienced higher rates of unemployment during the Great Recession 
(ibid.). 


Racial and ethnic minorities have faced greater barriers to post-secondary education and 
suffered greater economic disadvantages as a consequence. In 2013, some 24 percent of black 
and 21 percent of Hispanic young adults ages 18-24 were neither in school nor working, 
compared with 14 percent of non-Hispanic whites. Black and Hispanic young adults are much 
less likely to receive a bachelor’s degree than their white counterparts and are more likely to 
attend college without receiving any kind of credential (Holzer and Dunlop 2013). Not 
surprisingly, the Great Recession hit young black men especially hard: 37 percent of those ages 
20-24 were disconnected in 2010 (Dennett and Modestino 2013). 
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The implications of educational disadvantages extend to many other domains. Low earnings put 
young adults at high risk of poverty and associated ills, including poor health, higher crime, 
stressful living conditions, early and non-marital childbearing, and transmission of poverty to 
successive generations (National Research Council 2014). 


Demand for Middle-Skilled Workers. Looking at projected demand for workers with some high 
school but less than a four-year degree in growing technical fields, Year Up and other workforce 
organizations see a “win-win” solution. By expanding investments in postsecondary education 
for disadvantaged adults, they have argued that society can expand opportunities while 
addressing skills shortages and fostering economic growth. 


Economists have actively debated trends in supply and demand for sub-baccalaureate-level 
workers. Some analysts have emphasized a “hollowing” of job opportunities for middle-skilled 
workers, citing evidence of slower growth compared to lower- and higher-skilled occupations 
(e.g., Autor 2010). Historically, such jobs were concentrated in traditional manufacturing, 
construction and production sectors and required routine manual and cognitive skills. While 
acknowledging slower growth in middle-skill jobs, other economists have noted that the 
number of such jobs is nonetheless large and growing faster than the supply of new workers 
(Holzer 2010). They have pointed also to a large segment of rapidly-growing, skilled technical 
jobs within the middle—jobs requiring some postsecondary education, but not a four-year 
degree. Analysts variously have labeled these as “new middle-skill” or “skilled technical” 
occupations to distinguish them from traditional blue collar and office jobs (Holzer 2015, 
National Research Council 2017). 


One recent study identified 272 growing occupations in which more than one-third of workers 
had some college but not a four-year degree (Modestino 2015). Jobs in the information 
technology (e.g., network administrator, computer support technician) and health care 

(e.g., technician and therapy specialties) sectors were prominent among these occupations. 
Overall, these occupations accounted for a third of all U.S. workers in 2012. The study 
forecasted a shortfall of 3.4 million workers in such occupations by 2022. 


Such forecasts do not account for shifts in education requirements due to longer-term 
technological change and shorter-term business cycles. Though the former are difficult to 
project, recent evidence suggests how employers will respond to flux in economic conditions. 


The Great Recession accelerated growth in vacancies requiring bachelor’s degrees for jobs that 
had not previously required them—a phenomenon known as “up-skilling” (Holzer 2015). 
Analysts have attributed this surge to employers’ increased ability in weak labor markets to hire 
college graduates without raising wages (Holzer 2015) and the convenience of relying on a 
bachelor’s degree in lieu of direct screening for needed skills (Burning Glass 2014). 


More recently, the evidence suggests that the reverse has occurred—so-called “down-skilling” 
as labor markets have tightened, putting upward pressure on wages. A recent analysis of 

82.5 million online job postings found strong signs of such an effect: between 2010 and 2014, a 
one-percentage point reduction in the local unemployment rate was associated with a 0.27- 
percentage point reduction in the fraction of jobs requiring at least a bachelor’s degree 
(Modestino et al. 2016). In surveys, large numbers of employers cite difficulties in finding 
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workers with suitable technical and “soft” skills—particularly in growing sectors such as health 
care, information technology, and advanced manufacturing (Accenture et al. 2014). 


The evidence thus suggests growth in skilled technical jobs accessible with a year or two of 
post-secondary training and willingness of employers to adjust position requirements in 
response to shifts in supply and demand. But employers’ willingness to hire for skilled technical 
positions at this level will depend on their ability to find strong candidates at a reasonable cost. 


That is the demand that Year Up and other organizations seek to activate in recruiting and 
training low-income youth and adults. The strong emphasis on the demand, as well as supply, 
side of the labor market is a shift from older employment and training programs. As 
summarized in the next section, the newer “sectoral” approaches are showing greater promise 
than past approaches. 


1.3 Evaluating Year Up in the Context of Prior Random Assignment Studies 


Earlier generations of workforce programs targeted a wide cross-section of disadvantaged 
youth and generally did not train for fast-growing technical occupations. Newer interventions 
target somewhat less disadvantaged segments of the low-income population and intensive 
training in high-demand occupational sectors. 


As context for the Year Up evaluation, Section 1.3.1 traces findings from RCT evaluations of 
successive generations of workforce programs. Section 1.3.2 compares Year Up’s approach with 
that of other promising sectoral training programs and summarizes findings from a small 
previous Year Up study. 


1.3.1 Evidence on Other Workforce Programs for Young Adults 


The earliest national programs providing short-term training and supports to economically 
disadvantaged youth populations were mostly unsuccessful. Findings for the Job Training and 
Partnership Act showing no positive impacts for out-of-school 16- to 21-year-olds (Bloom et al. 
1997) prompted cuts in program funding for young adults. JOBSTART, a 13-site demonstration 
targeting high school dropouts aged 17-21 also found little impact on labor-market outcomes in 
all but one site (Cave et al. 1993).*° 


A second set of demonstrations involved more intensive services and had more positive effects. 
The most successful of these programs, National Guard Youth ChalleNGe, is a 22-week 
residential program for 16- to 18-year-old high school dropouts (Millenky et al. 2011). It 
provides education and training services within a disciplined, quasi-military environment and 
extends follow-up mentoring for 12 months after completion. In addition to positive impacts on 
GED receipt and postsecondary enrollment, a 10-site RCT found that ChalleNGe increased 
earnings by $2,266—a 20-percent effect—in the third year after random assignment (the first 
full post-program follow-up year). 


io Although this site—the Center on Employment and Training (CET) in San Jose—also proved effective in a 


demonstration targeting single parents (Zambrowski and Gordon 1993), the CET model proved difficult to 
replicate. A 12-site replication found little impact on employment and earnings for out-of-school 16- to 21-year- 
olds (Miller et al. 2005). 
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Job Corps, another residential program, also focuses on relatively disadvantaged 16- to 24-year- 
olds: about three-quarters of sample members in a nationwide evaluation were high school 
dropouts (Schochet et al. 2008). The program boosted receipt of GEDs and vocational 
certifications. It increased earnings by as much as 10 percent in the fourth follow-up year, with 
the largest effects for 20- to 24-year-olds. However, impacts faded over the next year or two. 


A number of studies tested youth-focused programs emphasizing work experience. At the more 
intensive end, YouthBuild provides education and training mainly in construction and hands-on 
experience with home building in community service projects along with supports. It targets 16- 
to 24-year-old high school dropouts and screens for “mental toughness.” YouthBuild produced 
modest increases in GED receipt and postsecondary enrollment but only small earnings impacts 
over an initial two-and-half-year follow-up period (Miller et al. 2016).** Tests of two national 
community service programs for out-of-school 18- to 25-year-olds, which also involved light 
training and supports, found little (Jastrzab et al. 1996) or no (Price et al. 2011) positive labor- 
market impacts. A “light touch” program in New York City—the Young Adult Internship 
Program—provides paid internships lasting 10-12 weeks in varied sites to 16- to 24-year-olds 
who are neither in school nor working. Early findings show modest positive earnings impacts 
($182, a 13-percent increase) in the fourth follow-up quarter (Skemer et al. 2017). 


The most promising evidence to date is for programs that target more advantaged low-income 
populations—generally motivated adults with high school credentials—and provide intensive 
short-term training and supports for skilled technical jobs in growing occupational sectors. 


These “sectoral programs” focus on career-track employment in professional occupations such 
as health care, IT, business, and financial services, as well as more skill-intensive jobs in 
manufacturing and the trades (Conway & Giloth 2014, Maguire 2016). They take a “dual 
customer” approach, treating both employers and disadvantaged adults as clients. Sectoral 
programs aim to become a reliable and trusted source of qualified workers equipped with skills 
local employers need. In striving to address local labor demand as well as possible, they screen 
stringently for ability to benefit from program services. A typical program accepts only one in 
five applicants (e.g., Hendra et al. 2016). 


Sectoral programs evaluated to date have operated on a small scale in localized occupational 
niches. The first RCT study, the Sectoral Employment Impact Study (SEIS), tested programs 
operated by three experienced providers (Maguire et al. 2010). The programs targeted low- 
income adults with weak employment outcomes and screened on basic academic skills and 
motivation. Training varied from two to 22 weeks and variously targeted jobs in IT, health care, 
office administration, construction, and manufacturing. All three programs had substantial 
positive impacts on earnings: together, they increased earnings by $4,011 (29 percent) in the 
second follow-up year. Across the SEIS sites, subgroup analysis showed a sizable $3,092 

(25 percent) impact for a subsample of 281 young adults’ ages 18-24. Impacts varied somewhat 


* A430-month follow-up survey found a statistically significant $16 increase in weekly earnings (a 12 percent 


effect), whereas estimates based on wage records data were close to zero and statistically insignificant. 


Abt Associates 1. Introduction 1 pg. 6 


Year Up 
Implementation and Early Impact Report 
PACE 


across programs. The largest effects were for an IT training program operated by Per Scholas, 
which generated a $4,663 (32 percent) impact.’” 


The SEIS findings prompted a follow-on demonstration, WorkAdvance, to assess whether 
sectoral programs could be effectively implemented by organizations with little prior 
experience (Hendra et al. 2016). The basic model was similar to SEIS but added job retention 
and advancement services to support next steps on career ladders. Three of the four 
participating organizations developed new programs for the study. The fourth, Per Scholas, 
provided an opportunity to see whether previous results could be replicated. 


The overall impact for the pooled WorkAdvance sample was positive in the second year, 
albeit—at $1,945 (14 percent)—substantially smaller than the SEIS impact. Smaller impacts at 
the three inexperienced WorkAdvance sites accounted for most of the difference. Per Scholas’s 
impact was again large ($3,746, or 26 percent), albeit somewhat smaller than in SEIS. Pooled 
across the four sites, subgroup analysis for young adults found positive impacts nearly identical 
to those for the full sample. 


The most recent addition to evidence on sectoral programs is an RCT evaluation of Project 
QUEST in San Antonio (Elliott and Roder 2017).This program seeks to engage and support low- 
income adults with a high school degree or equivalent in existing college programs providing 
training in health care occupations. Services include financial assistance with college expenses, 
remedial instruction, intensive individual and group counseling, and job search and placement 
services. A small RCT found that the program raised average earnings by $2,286 (14 percent) in 
the third follow-up year and that effects grew steadily to reach $5,080 (22 percent) by the sixth 
year. Unlike SEIS and WorkAdvance, however, impacts were not positive for 18- to 24-year- 
olds.*° 


1.3.2 Strengthening the Evidence Base: Evaluating Year Up in PACE 


Like other sectoral programs, Year Up incorporates stringent screening, training targeted to 
jobs in growing fields, intensive supports, work-based learning, and strong relationships with 
employers. But Year Up is unique in a number of respects. It is the only national multi-site 
sectoral program for young adults operated by a single nonprofit agency. At $28,290 per 
participant, it is a relatively intensive and costly program. Literature reviewed for this report 
offers no examples of professional internship programs for low-income young adults on a 
comparable scale and generating substantial revenue from employers. 


An earlier RCT found large earnings impacts for Year Up (Roder and Elliot 2014). However, the 
study sample was very small (102 treatment and 41 control group members at three local 


me ips interesting to note that, like Year Up, Per Scholas provides training and internships in entry-level positions 


in IT. While Year Up places interns with outside employers, Per Scholas interns work in the organization’s own 
revenue-generating computer refurbishing business. Although both programs help prepare participants for 
industry certification exams, such training is a more central feature of the Per Scholas than of the Year Up 


model (see Chapter 4, Section 4.2). 


*° The point estimates actually showed a negative impact for young adults and a positive impact for older adults. 


Notwithstanding the study’s small sample (100 of 341 total sample members were young adults), the 
difference in impacts was statistically significant (at p<.05; Elliot and Roder 2017, 14). 
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offices), and the design was flawed by a premature end to the embargo on services to control 
group members.’ Adjusting for this “crossover” from control to program conditions, the study 
found annual earnings impacts of $4,135 (34 percent) in the second year after random 
assignment, $4,934 (37 percent) in the third year, and $3,278 (24 percent) in the fourth year. 
There were large increases in employment in targeted occupational sectors (IT and financial 
services). Year Up’s strong emphasis on employment appears to have competed with college: 
rates of college enrollment were lower for treatment than for control group members after 
Year Up. 


The PACE evaluation of Year Up embodies a stronger design, with a much larger sample, a 
three-year embargo on services to randomly assigned control group members, richer data 
sources, and plans for long-term follow-up. The larger sample allows the study to detect smaller 
overall impacts as well as test for differences in impacts across subgroups. Subgroup analysis is 
valuable in gauging the implication of screening on varying characteristics during admissions. 
The availability of administrative as well as survey data extends the study’s ability to provide 
accurate measures of key outcomes for the full sample. Plans to follow up over a relatively long 
period and conduct a benefit-cost analysis mean the study will be able to assess how impacts 
evolve over a significant portion of participants’ lives and whether benefits outweigh costs. The 
PACE study also provides an opportunity to measure Year Up’s effectiveness in a strengthening 
economy. 


1.4 Organization of this Report 


This report has seven chapters. Chapter 2 provides an overview of Year Up’s organization and 
services. Chapter 3 describes the theory of change underlying Year Up and sets this theory 
within the career pathways framework guiding the PACE evaluation. It also documents how the 
study implemented random assignment and describes its principle data sources. Chapter 4 
analyzes program implementation. Chapter 5 assesses differences in experiences with 
employment and training and related services in the treatment and control groups. Chapter 6 
provides the report’s main findings on Year Up’s early impacts on employment, earnings, and 
career progress; postsecondary education; and other outcomes for the full sample and by 
subgroup. Chapter 7 summarizes and discusses the implications of the report’s findings. 


” Of the 41 control group members, 11 returned to participate in Year Up during the second and third years 


after random assignment. The authors used a standard approach to adjust for crossover, dividing the 
unadjusted impact estimates by the difference between the percentages of treatment and control group 
members who ever participated. The resulting estimates represent Year Up’s impacts for those who 
participated (in this case, 88 percent of treatment group members), rather than the total sample randomly 
assigned. 
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2. The Year Up Program 


This chapter describes Year Up’s local contexts and its major organizational and programmatic 
features. It begins in Section 2.1 with an overview of the program model. Section 2.2 profiles 
the urban areas in which local programs operate. Section 2.3 describes each of four major 
categories of services Year Up provides: assessment (2.3.1), instruction (2.3.2), supports (2.3.3), 
and employment connections (2.3.4). 


2.1 Overview 


In 2000, Gerald Chertavian founded Year Up to serve “disconnected” young adults ages 18 to 
24—youth with a high school diploma or equivalent, but who are neither employed nor in 
school full-time. ’® Year Up designed its model to meet employers’ needs for entry-level workers 
by preparing young adults from economically disadvantaged backgrounds with relevant 
technical and professional skills. The program teaches students high-demand technical and 
professional skills, connects them with employers for internships, and provides college credits 
via agreements with a local college partner. 


Year Up’s original program—its stand-alone “core” program—operates from eight offices in 
major U.S. cities.”? A national staff team headquartered in Boston supports local offices with 
operations assistance (such as human resources, marketing, and accounting); facilitates cross- 
office knowledge building; and provides guidance and oversight on implementation. Each Year 
Up office is in or near the central business district of the city in which it operates. Year Up 
receives funding from employer payments for interns; foundation grants; donations from 
companies and individuals; and, to a small degree, government agencies. 


Year Up seeks to recruit a diverse group of low-income students who are motivated and who, 
with assistance, can overcome challenges or barriers that may be affecting their ability to 
pursue career-track employment. Though students may have had some prior college 
experience, most Year Up offices do not accept applicants with bachelor’s degrees,”° 


The program enrolls participants in small groups of about 40 students, called “learning 
communities,” in March and September of each year. Entering cohorts in most offices include 
two or three learning communities. The program operates in two six-month phases: 


e During the first half of the program—the Learning and Development (L&D) Phase— 
students attend classes at Year Up from 8:30 AM until 3:30 PM four days a week, and 
for a half day on Wednesdays.” Training addresses both occupation-specific and 
general skills. The focus of technical training varies by office and cohort among five 


*8 For more information on Year Up’s development, see Chertavian (2012). 


*° The PACE evaluation is studying Year Up’s original core program. The organization also has developed a newer 


version of its model—the Professional Training Corps—that is now operating in over 15 college settings. 


a Applicants with an associate’s degree generally are eligible. 


2) The Wednesday schedule gives L&D students an unstructured afternoon when they can catch up on 


homework and other life needs. It also frees up staff and space for Wednesday afternoon sessions with 
returning interns (see below). 
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target occupational fields: IT (the most common), quality assurance, financial 
operations, project management, and customer service. Other courses focus on 
professional and business communications skills. Students gain experience in writing, 
giving presentations, interacting with clients and colleagues, and developing critical- 
thinking skills. Exhibit 2-1 summarizes the key program features of each local office. 


e Inthe second half of the program—the Internship Phase—students work in entry-level 
professional roles at local employers, often major corporations. They maintain full-time 
schedules, working at internship sites four and a half days a week and returning to Year 
Up on Wednesday afternoons to share and process internship experiences, attend 
workshops, and plan post-program career transitions. 


Exhibit 2-1: Selected Characteristics of Local Year Up Programs 


Maximum College Credits 


Office! Students per Occupational Partner Awarded for Participation in 
Year of Inception Cohort Focus College Year Up 
Atlanta, GA 85 Information Technology Atlanta Metropolitan 21 credits + IT Certificate 
2009 College 
Bay Area, CA 80 (San Information Technology — Foothill College 29 quarter semester credits + 
(San Francisco 2008, Francisco) Quality Assurance Fundamentals of Technical 
San Jose 2012) 40 (San Jose) Support Certificate 
Boston, MA 180 Information Technology © Cambridge College 18 credits 
2001 Financial Operations 
Chicago, IL 80 Information Technology Harold Washington 18 credits 
2010 College 
Washington, DC 120 Information Technology —_ Northern Virginia 18 credits 
2006 Community 
College 
New York City 135 Information Technology SUNY Empire State 30+ credits (18-21 credits per 
2006 Quality Assurance College track, with additional credits 
Financial Operations possible for submitting an 
Project Management internship portfolio) 
Providence, RI 80 Information Technology © Community College 23 credits for IT; 22 for 
2005 Customer Service of Rhode Island Customer Service 
Puget Sound, WA 80 Information Technology —_ Bellevue College 18 credits + Intro to 
(Greater Seattle Area) Quality Assurance Information Technology 
2011 Certificate 


NOTE: Quarter semester credits at Foothill College represent about 19 regular semester hours, following the general standard that 
each quarter semester credit represents .6667 semester credits. 


Employers are heavily involved in the program. They provide input on the program design, 
participate in program activities with students, teach workshops, provide internships, hire a 
substantial number of program graduates, and help to keep staff and students up to date on 
the latest developments and needs in relevant occupational sectors. Also, as noted in 

Section 2.3.4, employer payments for interns represent a major source of revenue for Year Up. 
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2.2 Local Context 


Before describing the program in more detail, this section provides background on the nine 
urban settings that host Year Up’s core (i.e., stand-alone) program offices. The metropolitan 
areas include Atlanta; Boston; Chicago; New York; Providence; San Francisco; San Jose; Seattle; 
and Washington, DC. The program operates from a single central location in each city, with for 
the exception of the two Bay Area offices. ** The first section below summarizes basic 
demographic and economic statistics for the nine metropolitan areas. The second section 
briefly notes other services that are available to young adults in these areas. 


2.2.1 Population, Geography, and Labor Markets 


The nine urban areas span both coasts and the Midwest. Census Bureau estimates for 
Metropolitan Statistical Areas (MSAs) in 2013—the year that Year Up began enrollment for 
PACE—indicate widely-varying demographic and economic characteristics (see Exhibit 2-2). 


Total populations range from under two million (Providence and San Jose) to nearly 20 million 
(New York) residents. African Americans and Latinos comprise a substantial proportion of the 
population in all but three areas (Boston, Providence, and Seattle). The proportion of adults 
with at least a bachelor’s degree varies from 30 percent (Providence) to 50 percent 
(Washington, DC). 


Poverty rates range from 9 percent (Washington, DC) to 16 percent (Atlanta), with rates of cash 
welfare and food stamps receipt generally tracking poverty levels. The “disconnection rate” for 
young adults—measured here as the fraction of 16-24-year olds not currently in school or 
working—varies from 8 percent (Boston) to 15 percent (Atlanta) and also generally tracks 
poverty. 


In most of the Year Up MSAs, the proportion of all jobs in computer and mathematical 
occupations and business and financial operations is higher than for the US as a whole. 
Estimates from the Department of Labor’s Bureau of Labor Statistics show higher 
concentrations of jobs in these target sectors in all but one MSA (Providence). In four MSAs 
(San Francisco, San Jose, Seattle, and Washington, DC), the proportion of all workers employed 
in computer and mathematical occupations were two to four times higher than for the nation 
as a whole. 


Year Up explicitly trains for financial operations occupations in two locations—Boston and New 
York—but its IT training also provides skills useful in a wide range of business and financial 
services occupations. Concentrations in this sector are higher for most of the Year Up MSAs 
than for the nation as a whole, albeit to a somewhat lower degree than for computer and 
mathematical occupations. 


Employment growth in these occupational sectors from 2013-2016 (the follow-up period for 
this report) provides a rough indication of the degree to which opportunities were expanding 
during the period covered by follow-up data on PACE sample members in this report. Again, 


2 Year Up administers its San Francisco and San Jose offices jointly and treats them as a single office in statistical 


reports. 
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conditions were quite variable. Employment grew rapidly in both sectors in New York, San 
Francisco, and San Jose and very little in Boston, Chicago and Washington, DC. 


A final statistic summarizes the availability of mass transit in each MSA, a critical support for 
many disadvantaged young adults in commuting to school and work. Index scores indicate 
substantially better access in some MSAs (e.g., New York and San Francisco) than others 
(Atlanta, Providence, and San Jose). These statistics show that Year Up’s nine offices operate in 
varied environments. An implication is that PACE provides a strong test of the degree to which 
the program model can be implemented successfully and generate its intended impacts in 
diverse urban settings. 


2.2.2 Other Services Available in the Community 


Each of the metropolitan areas offers an extensive range of alternative employment and 
training services, including training at community and technical colleges, for-profit 
postsecondary institutions, and non-profit training providers. Some of the better known 
community-based providers include Job Corps, Per Scholas, Jewish Vocational Services, Center 
on Employment and Training, and Goodwill Industries. Many training providers offer job 
readiness and job placement services in conjunction with technical skills training. Some offer 
participants mentoring by volunteers in the business community. 


At some providers, training is offered to participants free-of-charge. However, for many of 
these education or training options, individuals likely would need to navigate financial aid or 
identify other sources of funding. In some communities, individuals might qualify for college 
access programs that provide free financial aid advising. A few programs offer paid internships 
or work experience opportunities (e.g., Per Scholas in Boston and the Young Adult Internship 
Program in New York). Few programs offer extended professional experience at major 
corporations comparable to Year Up’s internships. 


2.3 Local Office Staffing 
Year Up local offices maintained five basic categories of staff during PACE: 
e Admissions and student services (typically located in the same department); 


e Program and academics (including learning community leaders and instructors, also 
often located together); 


e Corporate partnerships (typically including employer partner development, internship 
services, and career and alumni services); 


e Development (sometimes including volunteer coordination); and 


e Operations (typically including accounting, human resources, facilities and related 
strategic planning). 


An executive director in each office provided overall leadership and devoted substantial 
attention to external relationships and fundraising. Most offices also had a deputy with more 
responsibility for day-to-day management: this role (also known as site director or program 
director) became more common over the study period. A typical office had about 40 regular 
staff. 
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Exhibit 2-2: Characteristics of Metropolitan Statistical Areas with Year Up Core Offices 
New San Washington, 
Characteristic Atlanta Boston Chicago York Providence Francisco SanJose _ Seattle DC 

Total population, 2013 (1,000s) 5,525 4,684 9,537 19,950 1,604 4,516 1,920 3,610 5,950 
Disconnected 16-24 year olds in 20132 

Percent of all youth 14.9 8.2 12.5 13.5 13.0 10.4 9.7 12.4 12.4 

Number (1,000s) 111 49 148 324 28 51 20 51 94 
Black or African American (%) 33.0 77 16.9 17.2 5.3 7.9 2.7 5.7 25.5 
Hispanic or Latino Origin, Any Race (%) 10.5 9.9 21.4 23.5 11.2 21.9 27.7 9.5 14.7 
Educational Attainment, 25+ year olds (%) 

Less than high school graduate 11.8 8.8 12.8 14.7 15.2 11.9 13.5 8.3 9.5 

High school graduate/equivalent 24.8 24.0 24.8 25.6 28.1 16.9 15.3 20.3 19.1 

Some college or associate’s degree 28.2 22.4 27.2 22.2 27.1 26.1 24.5 31.9 22.8 

Bachelor's degree 22.4 25.0 21.4 21.9 18.2 27.0 25.3 25.2 25.3 

Graduate or professional degree 12.7 19.8 13.7 15.5 11.4 18.2 21.3 14.2 23.3 
Unemployed (%) 9.8 12 9.8 8.8 9.6 8.0 8.3 7.3 6.7 
Below Poverty (%) 15.9 10.4 14.4 14.6 14.3 11.5 10.5 12.6 8.5 
In Household with Benefits in Last 12 Months from 

Cash public assistance 2:1 2.9 27 3.2 3.2 2.8 2.3 3.4 2.0 

Food Stamp/SNAP benefits 14.2 11.3 13.3 14.0 16.2 6.3 5.4 12.6 7.9 
Prominence of Occupation (Location Quotient)® 

Computer and mathematical occupations 1.44 1.77 1.08 1.08 0.89 1.91 3.76 2.50 2.72 

Business and financial operations 1.36 1.33 1.08 1.22 0.93 1.46 1.38 1.38 2.06 
Change in Number Employed 2013-16 (%) 

Computer and mathematical occupations 23.5 6.6 13.7 13.8 13.0 24.2 34.0 5.2 0.5 

Business and financial operations 6.0 -0.2 11.4 11.0 9.8 18.3 18.2 17.9 0.4 
Extent of Transit Index (0-100)¢ 44 74 65 84 48 80 41 57 71 


SOURCES: 2013 population characteristics at www.factfinder.census.gov, disconnection estimates from Lewis & Burd-Sharps (2015), occupation statistics at www.bls.gov/oes/tables, 
transit scores at www.walkscore.com/methodology.shtml. 

NOTES: “Estimates of disconnection are based on current school and work status. ’ The “location quotient” is a Bureau of Labor Statistics measure representing the ratio of the 
fraction of all workers in a particular occupation in the MSA to the fraction for the US overall. Values above one indicate that an occupation is more prominent in the MSA than in the 
nation as a whole. © Higher values on the transit index indicate greater access to major transit routes (factors include frequency, transit mode, and distances to nearest stop). 
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In addition to staff, each office appointed an executive board of leaders from local business, 
philanthropic, and education communities. These boards provided ongoing counsel on strategy 
and developments in the community and often played key roles in introducing Year Up leaders 
to new donors, corporate partners and college partners. 


2.4 Program Design 


This section describes Year Up’s major services, organized under the four major career 
pathways framework components (Fein 2012): assessment (2.4.1), instruction (2.4.2), supports 
(2.4.3), and employment connections (2.4.4). 


2.4.1 Assessment 


A comprehensive approach to assessing participants and program performance is a hallmark of 
promising career pathways programs. The Year Up model includes several basic assessment 
components, each directed to a different goal. Two components guide decisions and supports 
pertaining to individual applicants and participants, while a third component involves 
monitoring and supporting high performance at the local office level. 


First, to identify and select applicants who are a good fit to the program, admissions staff in 
each local office administer an intensive, multi-stage assessment and screening process. 
Recruitment and screening is ongoing, timed to generate target number of admissions to 
cohorts starting the program in March and September of each year. Key steps include: 


1. Online interest form. Potential applicants complete a student interest form on Year 
Up’s website. The form covers basic demographic, educational, and employment 
background information. 


2. Information session. Young adults then attend an information session where they 
receive a tour of the local Year Up office and an overview of program. 


3. Application. If interested, they then submit an application, attaching a resume, a two- 
page essay describing their interest in Year Up, and references. 


4. Drug screens and background checks. Local offices conduct drug screens and 
background checks at varying points in the process. Some offices screen up front and 
condition admission on passing drug tests, while others test after acceptance to identify 
needed support services. 


5. Learning assessment. The next step involves assessing applicants’ abilities and skills 
through individual and group activities. Although formats and activities vary, most 
offices assess reading, writing, and critical-thinking abilities, and some offices assess 
math abilities. Learning assessments generally included a public-speaking activity and 
interactive group activities to evaluate attitudes, motivation, and interpersonal skills. 


6. One-on-one interview. Applicants who pass the learning assessment screen then have 
one-on-one interviews with program staff. Interviews focus on strengths, motivation, 
career interests, reasons for applying to the program, and a series of personal factors 
that might affect success in Year Up (for example, stable housing, transportation, and 
child care). Using a standardized rubric, interviewers assign “risk” scores to capture the 
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existence and severity of each of a set of potential barriers. Admission teams tally scores 
across barriers for a summary measure of risk. 


Decision forum. Towards the end of the six-month recruitment period for each 
incoming cohort, admissions teams meet to review and discuss each applicant’s 
strengths and weaknesses and make acceptance decisions. The reviews focus on each 
candidate’s ability to meet the program’s demands, given available supports, and the 
likelihood that they will benefit from participating. The goal of the process is to identify 
applicants who can thrive in Year Up but who otherwise might not succeed in starting a 
career. Applied to summary risk scores, for example, program guidelines encourage 
local staff to focus on candidates in the intermediate range—rather than those in the 
lowest or highest ranges. 


In addition to supporting admissions decisions, local staff use information gathered 
through the above steps to identify and arrange needed supports for accepted 
applicants. 


A second aspect of Year Up’s comprehensive approach to assessment is ongoing monitoring of 
each participant’s progress during the L&D and internship phases: 


During the L&D Phase, in addition to graded classroom assignments, students receive 
feedback from seven- and 14-week assessments, styled after performance reviews in 
the corporate world. Approaches vary by office, but tend to include feedback from 
instructors, advisors, and learning community directors. The aim is to identify whether a 
student is “on track” or “not on track” in anumber of domains, from course 
performance to demonstrated independence. The 14-week assessment also highlights 
areas in which the student has improved since the previous assessment. The goal of 
these assessments is to help students identify additional steps needed to successfully 
complete the L&D phase and earn an internship. As another feedback mechanism, some 
offices work with certain students to develop “hire-ready action plans” that outline 
tangible steps in preparing to make a strong impression on potential employers. 


During the Internship Phase, Year Up internship specialists monitor interns by email, 
phone calls, and site visits with interns and work supervisors. Work supervisors also 
provide formal feedback on student attendance, dependability, initiative, technical 
performance, and professional demeanor via online surveys at two time points during 
the internship (weeks 8-10 and weeks 18-20). Supervisors assign ratings and provide 
written comments on each intern’s strengths, weaknesses, and job prospects. They also 
file weekly reports on attendance and timeliness. Finally, the program solicits input from 
host employers on areas in which Year Up can strengthen its curriculum to better meet 
their business needs. 


The third pillar in Year Up’s approach to assessment focus involves monitoring local office 
performance. A key tool is a set of quantitative indicators known as “FM-RADIO.” Drawing from 
Year Up’s management information system (based on a Salesforce® database), national and 
local staff monitor weekly dashboard reports for each office summarizing performance on: 
Financial Management (revenue covers costs), student Retention, Admissions (meeting 
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recruitment targets), Development (fundraising), Internships (securing enough positions), and 
post-graduation employment Outcomes. Local offices are expected to meet performance 
targets in each area. Illustrative targets included achieving 75 percent program retention to 
graduation, developing internship placements for all L&D completers, and ensuring that at least 
70 percent of participants were working full-time in Year Up target occupations four months 
after graduating. Failure to meet targets typically prompts intervention by national office staff, 
to help troubleshoot and develop responses to problems. 


2.4.2 Instruction 


During L&D, Year Up offers courses in three subjects. This section describes the three subject 
areas, as well as arrangements with local colleges to grant credit for Year Up courses. 


One set of courses focuses on technical skills in specified occupational tracks. Occupational 
tracks vary by office (see Exhibit 2-1). Within tracks, offices have the flexibility to choose or 
develop curricula to meet the needs of local employer and generate credit through 
arrangements with local colleges (described below). Some offices use existing curriculum, such 
as Cisco’s IT Essentials for those in the IT track, whereas others create their own curricula with 
input from employers and assistance from consultants. 


A second course—Business Communications—focuses on English and related communication 
skills needed to succeed in corporate settings. In these courses, students learn how to 
compose, analyze, and respond to professional emails. Instruction in basic reading, writing, and 
critical-thinking abilities draws heavily on situations and tasks that arise in professional 
environments. Students also learn public-speaking techniques and how to express ideas 
persuasively, as well as develop an “elevator pitch” about their career interests and skills. 
Business Communications introduces the concept of code switching—or being able to move 
alternately between two different languages or dialects. This concept encourages students to 
become fluent in the language and culture of corporate America, while acknowledging that 
different manners and vocabulary may be appropriate in students’ home communities. 


A third course—Pro Skills—focuses on the wider set of expectations and behaviors needed to 
function effectively in professional settings. Classes explore business etiquette, non-verbal 
communication, workplace relationship skills, and appropriate responses to common workplace 
scenarios. To help students manage their lives outside of the classroom or workplace, Pro Skills 
courses also typically include sessions on financial literacy, including topics such as debt, 
managing credit, and budgeting. 


During the PACE period, local offices had substantial flexibility to tailor curricula to local 
circumstances. Exhibit 2-3 gives a sense of some of the ways local approaches varied to reflect 
the interests and experience of local instructors, curriculum standards of local college partners, 
and skills prioritized by area employers. 
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Exhibit 2-3: Local Flexibility in Curriculum Development 


During PACE (2013-2014), Year Up’s national office outlined learning objectives that all offices should cover in specified 
courses, but it did not provide standardized curricula. The philosophy was that this flexibility would lead to better 
customization to the curriculum standards of local college partners, to skills employers needed, and to the teaching strengths 
and interests of individual instructors. 


Local offices sometimes borrowed curricula from other Year Up offices or outside organizations, and they sometimes 
developed curricula from scratch. For example, one office modified Cisco’s IT Essentials curriculum to include different 
examples and activities and meet its college partner's standards. Another office updated its college partner's curriculum to 
reflect recent changes in the IT industry. Yet another office developed an entire track in financial operations working closely 
with a large partner employer. 


Year Up’s Business Communications courses varied across the eight offices. In most offices, these courses were highly 
applied—focusing on language and style in emails and other workplace-relevant communications. Other offices took a more 
traditional academic approach, sometimes to qualify for English credit from the college partner and sometimes because staff 
believed such an approach had more general benefits. The traditional version might emphasize mechanics, structure, and 
style in formal essay writing or bring more literature into reading assignments. 


Curricula for Year Up’s Professional Skills course were more consistent across offices. These courses typically covered 
practical business skills (e.g., handshakes, eye contact, small talk, body language, dress code); networking and personal 
“branding”, and preparing for internships (e.g., what to expect; interpersonal conflict; managing your manager). Offices 
variously put more and less emphasis on certain skills, such as critical thinking, problem solving, and financial literacy. 


Towards the end of the PACE period, Year Up's national leaders took initial steps to centralize curriculum development. This 
step addressed the increasing difficulty of monitoring curriculum quality across an increasing number of locations and 
occupational tracks and the greater efficiency of centralized development. 


As part of the program model, each office established a formal agreement with a local college 
partner. Under these agreements, the colleges enrolled participants as students, granted credit 
for successful completion of courses, and sometimes awarded short-term certificates for 
completing Year Up. The level of support from Year Up staff in the enrollment process varied. In 
some offices staff guided students through each step of the enrollment process, including 
applying for financial aid and signing up for placement exams, whereas in others students were 
mainly left to apply on their own. Year Up handled the portion of tuition and fees not covered 
by Pell grants and other financial aid, making college available at no cost to participants. 


In tailoring some of its courses to meet local colleges’ requirements for credit, Year Up aims to 
foster progress towards college credentials in the longer-term while emphasizing career-track 

employment in the short-run. This dual focus—on articulation between successive training and 
employment steps—is a core career pathways principle (Fein 2012). 


While nearly all college partners granted credit for program’s technical and business courses, 
only a few awarded credit for Year Up’s Pro Skills course. In some offices, colleges granted 
English credit for Year Up’s Business Communications course only for students scoring high 
enough on college placement exams to qualify for college-level English. 


Although graduating from Year Up does not normally result in an occupational certificate, a few 
partner colleges award certificates to Year Up graduates for work completed in Year Up. 
Examples include certificates with titles such as Fundamentals of Technical Support and 
Introduction to Information Technology. 


Preparation for industry certification is not a core feature in Year Up’s technical courses. 
However, a number of offices provide ad hoc supports such as: 1-2 week training boot camps, 
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shorter workshops, elective courses taught by volunteers from local employers, and financial 
assistance with examination fees. 


In addition to formal classes, Year Up also offers multiple opportunities for participants to gain 
wider exposure to local professionals and practice work-relevant skills during L&D (see 
Exhibit 2-4). 


Exhibit 2-4: Fostering Career Knowledge Beyond the Classroom During L&D 


In addition to internships and a highly work-oriented, hands-on curriculum, Year Up provides its students a variety of 
Opportunities to experience professional situations during the L&D phase. For example, many students participate in Year Up 
marketing and outreach activities. Such activities offer them real-world occasions to practice their business communications 
and professional skills and to get to know local professionals. Visits from and to local employers familiarize students with 
technologies in use, work environments, and career pathways at different firms. In addition to building students’ knowledge 
and skills, these activities are one of the program’s most potent marketing tools. 


Student Ambassadors. Year Up generates interest in wider professional communities, including among local community 
organizations, policymakers, funding agencies, researchers, and businesses, as well as the media. One way it channels this 
interest is through office tours conducted by “student ambassadors.” During the tours, students describe program activities 
and answer visitors’ questions. Visits often include lunch with a group of students. Ambassador training for student tour 
leaders covers topics such as how to describe Year Up; share personal backgrounds, program experiences, and career 
aspirations; and ask visitors about their jobs, backgrounds, and companies. 


Outreach. Students can participate in program marketing events on and off site. Year Up events often include presentations 
from panels of Year Up students and alumni. One student interviewed for PACE described with pride how he was selected to 
give a speech at a Year Up golf tournament fundraiser and “talk about life in front of a bunch of people.” Several staff 
remarked that Year Up has learned that hearing students’ stories directly is one of the most effective tools for raising funds 
and developing partnerships with employers. 


Learning Community Activities. Another form of work-based learning is the opportunity to take responsibility for weekly 
learning community activities, such as Monday Morning Kick-Off and Friday Feedback. One office rotated students weekly in 
the manager role to plan and lead such activities. Another created a four-week training program in meeting facilitation for 
students interested in facilitating Friday Feedback. This training covered planning and structuring the sessions, as well as how 
to coach others in feedback skills. 


Employer Engagement. Year Up also exposes students to industry professionals by engaging employers as guest speakers, 
elective instructors, and participants in Year Up activities. For example, one office held “Tech Thursdays,” with weekly guest 
speakers sharing the latest developments in technology and giving tips on stepping into technology careers. Another office 
hosted sessions with employers on Fridays. Staff selected speakers whose life stories were likely to resonate—for example, 
one guest speaker from PayPal talked about what it was like to develop her career as a single parent. Students prepare for 
these sessions by researching the speaker's background and preparing questions for discussion. 


Workplace Simulations. Another often-used strategy involves simulated work situations. Such activities often involve on-site 
participation by employers. For example, one office held a “Tech Olympics” at the end of the L&D phase, with role playing in 
customer service scenarios and customer service situations. Employers played the roles of clients and end users. 


2.4.3. Supports 


Year Up espouses a philosophy of “high support, high expectations” that promotes high 
standards for both participant behavior and program services. As described below, a formal 
contract with each student lists expectations for professional behavior, and varied program 
activities encourage feedback on behavior among peers and staff. Learning communities aim to 
foster a supportive social environment. Other supports include staff social workers, advisors, 
mentors, financial assistance, and instructional supports. 
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Encouraging Professional Behavior 


Year Up specifies expected responsibilities and behaviors in a formal contract each student 
must sign as a condition of participating in the program. The contract sets forth expectations 
for attendance, being on time, professional dress, behavior and conduct, assignment 
completion, and respect for the core values of the program. One staff member described it as a 
tool that outlines the “rules to the game for corporate America.” 


Contract violations—called “infractions” —trigger loss of points and stipend reductions. 
Although details varied across offices, students generally start with 200 points. When points fall 
to a specified threshold (usually 100-150, depending on the office), students must work with 
staff on an improvement plan and present it to their learning community. If points reach zero, 
students are said to have “fired” themselves and must leave the program. 


Year Up stresses a high-feedback culture in which students learn to both accept and give 
feedback constructively. In “Friday Feedback” sessions, they gather with their learning 
community to give one another specific feedback using a structured method called the 
plus/delta approach.”° This method involves identifying positive aspects of recent experiences 
(the “plus”) and aspects they might wish to change (the “delta”). To underscore its philosophy 
of “high support, high expectations,” the program also encourages students to provide 
feedback to program staff. Friday Feedback sessions also are used to reinforce Year Up’s 
contract: point totals and infractions earned by students during the week are read aloud and 
discussed. Staff encourage students to apply skills feedback in all L&D activities as well as with 
co-workers and supervisors during internships. 


Social Supports 


Year Up’s social supports begin with orientation and extend beyond graduation. To foster a 
sense of belonging and help new students acclimate to the demands of Year Up, many offices 
offer pre-program activities. These activities include welcome receptions, “bring a supporter” 
events open to friends and family, question-and-answer sessions with the office director or 
other senior program staff, and “Gear Up for Year Up” sessions aimed at bolstering students’ 
engagement. 


As noted earlier, students are grouped in learning communities of about 40 students and 
designated staff members. Community members learn together and share a common Year Up 
identity. During the first week of the program, known as Orientation or “O-week,” learning 
community members engage in activities designed to build trust and cohesion. Activities 
emphasize “interactives” —small group exercises focused on key skills and principles that 
require active engagement and collaboration. Their purpose is to help students learn what they 
have in common with others, highlight unique background experiences, show the importance 
of supportive peer and career networks, establish shared norms for behavior, and hold one 
another accountable over the course of the program. 


Social Services 


Year Up’s Student Services staff includes trained social workers and mental health professionals 
equipped to support students’ efforts to negotiate challenges that may affect their chances of 


3 The plus/delta approach is a tool commonly used in facilitation, debriefing, and instructional environments. 
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completing the program. During admission, these staff members help to identify circumstances 
that can make it difficult for young adults to succeed in the program. Student services teams 
then help to decide whether the program has resources to support each individual and the 
number of higher-risk applicants the program can handle. Before classes start, social workers 
work with each admitted student to identify issues, develop an “onboarding plan” identifying 
needed preparatory steps, and arrange supports. For students needing more intensive 
supports, the program coordinates with a network of community-based providers to provide 
specialized services such as housing, child care, legal advice, medical insurance and care, and 
help accessing public benefits such as the Supplemental Nutrition Assistance Program (SNAP). 
Social workers participate in learning community activities over the course of L&D to get to 
know and be accessible to participants. 


Advising and Mentoring 


Each student is assigned a staff advisor at the outset of the program, and this relationship 
continues at least through graduation. Particularly during the L&D phase, advisors monitor each 
participant’s experiences in the program closely, providing encouragement and helping to solve 
problems that arise. All Year Up staff members, regardless of their role, serve as advisors to 
four to eight students. Advisors are required to meet with advisees on a weekly basis but have 
substantial flexibility in the format of these meetings. 


Year Up also pairs each student with an outside mentor from the community for additional 
career and personal guidance.~’ The program recruits mentors from the local business 
communities, often as part of the same outreach activities used to raise funds and develop 
internship positions at local firms. Mentors offer students an opportunity to meet and network 
with working professionals in settings relevant to their occupational interests. Mentors often 
participate in program activities such as tutoring and preparation for job search (e.g., resume 
development and practice interviews). At most offices, mentors are assigned after the students 
have been in the program for some time (e.g., midway through the L&D phase or early in the 
internship). Students and mentors meet once or twice a month, typically outside standard Year 
Up hours. Local offices provide training to mentors on Year Up and on expectations for the 
mentoring relationship. 


Financial Assistance 


Year Up provides its package of training and services at no cost to students. It also helps 
students address financial needs through weekly stipends and help applying for student 
financial aid. 


Stipends provide an opportunity to “earn while you learn” and are designed to help students 
meet needs that otherwise might prevent participation. In most offices, the weekly stipend was 
$150 during the L&D phase and $220 during the internship phase. In theory, a student who 
finished the program and received their full stipends each week would earn $8,870 for the 
entire program. Although less than they would receive in a full-time minimum-wage job 
($15,080 at the $7.25 federal minimum wage) the stipend’s purpose is to make it easier for 


** All but one office requires students to meet with their mentors. The remaining office thought that allowing 


students to decide if they wanted mentors would result in stronger mentoring relationships. 
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participants to work fewer hours in order to focus on the program, rather than provide a 
regular wage. 


Another function of stipends is to incentivize compliance with the expectations specified in the 
student contract. As explained above, contract infractions trigger reductions in stipends and 
exit from the program. Thus, the amount students actually receive may be less than the 
maximum possible. Chapter 4 reports data on infractions and actual stipend amounts received 
by the PACE cohorts. 


As mentioned above, local office staff help students assess eligibility and apply for Pell grants 
and other student financial aid to meet tuition and other school expenses required for 
enrollment at the programs’ local partner colleges. Year Up covers any remaining college costs 
not covered by this financial aid. 

Instructional Supports 

Local offices provide tutoring and other assistance to students who need additional academic 
help. The format and extent of such supports varies. In most offices, instructors or teaching 
assistants are available in computer labs after classes to assist students who have questions 
about course material. At least one office runs a peer tutoring program. 


2.4.4 Employment Connections 


Year Up provides students with actual employment experience during the internship phase of 
the program. It also provides intensive employment services towards the end of the program. 


As detailed in Chapter 4, Year Up has dedicated staff to develop internship positions at the 
national and local levels. In most instances, national leaders initiate and coordinate 
relationships with large national firms with workplaces in Year Up’s local markets. Corporate 
relations teams in local Year Up offices are responsible for developing internship positions— 
“internship sales” as the program calls it—as well as for maintaining ongoing relationships with 
local firms. 


Internships 

Following their coursework during the L&D phase, students continue their learning through six- 
month internships with companies in fields related to their program of study. The purpose of 
internships is to provide work experience, career awareness, and connections with potential 
future employers, while maintaining contact with Year Up staff and attending weekly meetings 
with staff and peers to “process” their work experiences. 


Employers pay Year Up for each intern to defray costs of recruiting, training, and supporting 
interns’ success. The average payment was $22,404 per intern in the PACE sample. Averaged 
across all participants (including those dropping out before internships), this revenue financed 
59 percent of the program’s $28,290 per participant cost. 


Several weeks before the internship phase begins, each local office runs an intensive operation 
to match interns with available internship slots. In matching students to internships, local 
offices consider student attributes such as personality, interests and abilities, outside life 
circumstances, location, and transportation options. They also consider key aspects of 
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internship positions including the work environment, supervisory styles, typical duties, future 
career possibilities, and location. 


During internships, students work full-time four days a week and a half-day on Wednesday. The 
somewhat higher stipend they receive during their internships (S220/week) helps to 
compensate for transportation and any other additional needs during this phase. Contract 
monitoring during internships focuses mainly on attendance and work hours, with worksite 
supervisors filing regular online reports. Otherwise, Internship Services staff focus on interns’ 
progress and any issues at work, rather than on enforcing other contract provisions. 


On Wednesdays, interns return to Year Up to debrief on internship experiences and reconnect 
with staff and peers. Activities vary across offices, but generally include advising, group 
discussion of progress and challenges in internships, and workshops on selected topics. 
Towards the end of internships, Wednesday activities emphasize career planning, resume 
development, and job search. 


Year Up encourages students to view internships as opportunities to gain hands-on experience, 
earn a strong letter of recommendation or reference, and potentially land a job after 
graduation from the program. 


Post-Program Services 

Each local Year Up office’s Career and Alumni Services (CAS) team assists participants with job 
search and placement after graduation. These services extend for up to four months. Services 
generally involve identifying pertinent job openings, sponsoring job fairs, and providing one-on- 
one support with career counseling, job search, and resume development. CAS teams staff an 
in-office resource center providing internet access and help completing online job applications 
and other job search activities. 


CAS teams also promote more extended engagement through local alumni boards and social 
media platforms such as LinkedIn and Facebook. Related opportunities for graduates are 
attending alumni events, speaking to current students at Year Up, and sponsoring internships at 
their place of work. 


In 2014, Year Up launched an affiliated employment services firm—Year Up Professional 
Resources (YUPRO)—dedicated to Year Up alumni. YUPRO provides career supports to alumni 
and fee-based recruitment services to employers. Its financial goals are to be self-sustaining, 
returning any profit to Year Up to support expanded alumni programming. 
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3. Evaluation of Year Up 


This chapter describes the evaluation design and analysis methods and data used in this report. 
Section 3.1 describes the theory of change underlying Year Up’s model. Section 3.2 outlines 
major research questions for the evaluation. Section 3.3 describes the evaluation’s research 
design, characteristics of the research sample, data, and analysis plan. 


3.1 Theory of Change for Year Up 


The PACE evaluation is using a career pathways framework (Fein 2012) to provide consistency 
in research questions and analyses across the nine programs in the study. This framework helps 
in identifying aspects of Year Up that are important to understand in evaluating the program’s 
implementation and impacts. 


Exhibit 3-1 depicts the PACE career pathways theory of change as applied to Year Up. It shows 
how major program components target intermediate outcomes, which in turn are expected to 
affect the main outcomes of interest when assessing whether the program was successful. 


3.1.1 Recruitment & Screening 


The Year Up program model begins with recruitment (see box A). As described in Chapter 2, 
recruitment targets urban young adults aged 18-24 who have a high school credential and who 
staff judge can benefit from the program. In assessing ability to benefit, admissions teams look 
for applicants who are motivated to succeed and whose challenges are manageable with 
supports the program can offer. 


3.1.2 Major Program Components 


Chapter 2 described the four major components of the Year Up program model. To recap, they 
include: 


1. Assessment. Year Up participants are assessed during the admissions process; during 
onboarding to the program; and throughout the program via course grades, periodic 
evaluations from staff and employers, and structured feedback. 


2. Instruction. Participants receive formal instruction in business communications; 
professional skills (e.g., practical business communication and networking skills, critical 
thinking and problem solving, teamwork, self-regulation, financial literacy); and technical 
skills (e.g., IT, financial operations). Local offices also provide a wide range of workshops 
and other ad hoc learning opportunities. 


3. Supports. Participants are offered a wide array of services to support successful 
completion of the program. These services include advising, mentorship, learning 
communities, supportive services and connections to external services, stipends, and 
contracts with financial incentives to encourage professional behavior. 


4. Employment Connections. During the latter half of the program, participants work as 
interns in professional settings and receive supports for career planning and post- 
program job placement. They also interact with employers in other aspects of the 
program. For example, during the L&D phase, employers teach workshops, participate in 
class projects and on-site events, and host workplace visits to introduce their companies. 
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Exhibit 3-1: Theory of Change for Year Up 


A. RECRUITMENT & SCREENING 


+ Ages 18-24, high school credential 
+ Motivation, manageable risks 
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E. CONTEXTUAL FACTORS 


+ Economic conditions and labor market context 
+ Institutional context for postsecondary education and training 
* Social context and socio-demographic profile of the young adult population 


* Business communications 
+ Professional skills 


+ Technical skills 2. PURSUIT OF FURTHER EDUCATION 


* College persistence 
* Career-relevant credentials 
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Intermediate Outcomes 


These program components variously aim to improve intermediate outcomes in five domains 
(see box C): 


1. General (21*-Century) Competencies. These competencies include cognitive skills 


3.1.4 


(e.g., literacy, numeracy, critical thinking, creativity); intrapersonal skills (e.g., core self- 
evaluation, work ethic/conscientiousness, self-regulation/meta-cognition, intellectual 
openness); and interpersonal skills (e.g., teamwork, collaboration, leadership). The Year 
Up program seeks to develop these skills through its Business Communications and Pro 
Skills courses, behavior contracts, learning communities and other social supports, and 
exposure to work situations during internships. 


Occupation-Specific Competencies. Technical courses; ad hoc learning opportunities at 
Year Up offices (e.g., workshops, elective courses, guest lectures); and hands-on 
experience during internships all impart specific occupational skills. 


Career Orientation and Knowledge. Many elements of the program aim to foster 
broader awareness of career options and pathways in the fields Year Up targets. They 
include extensive exposure to local employers, technical instruction from instructors 
with extensive industry experience, and formal career planning services. 


Material Resources. Stipends and supports arranged by student services teams are 
designed to alleviate material hardships that can interfere with students’ ability to 
participate in the program. 


Personal and Family Challenges. Close assessment and strong supports help to identify 
and address other potentially destabilizing life challenges. 


Main Outcomes 


Improvements in the intermediate outcomes are expected to improve three main outcome 
domains (see box D): 


1. Success in Career-Track Employment. Year Up’s theory of change defines success 


3.1.5 


following the program largely in terms of full-time work in targeted occupations at 
$15/hour or above at four months post-graduation. 


Pursuit of Further Education. The program also identifies college persistence and 
completion as important longer-term goals, once graduates are financially stable. To this 
end, during the program students are enrolled at local colleges and earn grant credits 
for completing Year Up courses. 


Improvements in Other Life Outcomes. The model posits that good jobs will raise 
income and assets and thus enhance other dimensions of individual and family well- 
being in the longer term (e.g., physical and mental health, formation of healthy family 
and social relationships). 


Contextual Factors 


Finally, each local Year Up program tailors its program to local conditions. These include the 
sociodemographic composition of young adults in the area, local industry and employment 
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base, economic conditions, and colleges and other post-secondary options in the community 
(see box E). 


3.2 Major Research Questions 


This report addresses a range of questions about Year Up’s implementation and early impacts. 
3.2.1 Questions about Implementation 


The implementation study assesses how program staff implemented the program and 
experiences operating it (Chapter 4) and what students experienced as participants (Chapter 5). 
Key questions include: 


e Where did the program operate, and what were the prevailing local economic and 
demographic characteristics? 


e What was the basic approach to staffing and operation in local offices, and what role did 
Year Up’s national office play? What basic services and strategies did local offices put 
into place? 


e How fully did Year Up implement the program model in each of the eight local offices? 


e To what degree did treatment group members progress through and complete the 
program’s two major phases? What were some of the challenges for participants and 
how did staff respond to those challenges? 


e To what extent did the services treatment group members experienced at Year Up differ 
from the services control group members received at other programs? 


3.2.2 Questions about Impacts 
The impact study analyzes impacts on intermediate and main outcomes (Chapter 6) over 


follow-up periods ranging from 1.5 to 2.5 years (depending on the data source). It addresses 
questions such as: 


e Did the program have impacts on students’ earnings in the period following Year Up 
post-graduation services? Did effects extend beyond that period? 


e Did the program have impacts on students’ employment in technical fields that Year Up 
targets? Did it affect their hourly wages? Foster a sense of career progress and 
supportive career networks? 


e Did connections with local college partners affect college enrollment during and after 
the period that treatment group members were completing the program? Were there 
impacts on credits earned and credentials received? 


e Did Year Up have impacts on financial hardship and strain? 


e Did the program affect other outcomes, such as psycho-social skills and resources (e.g., 
grit, academic self-confidence, and core self-evaluation) and family formation? 


e To what degree do impacts vary for students with different characteristics and in 
different Year Up local offices? 
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The impact analysis covers a relatively short follow-up period. Later reports will cover longer- 
term impacts. 


3.3 Evaluation Design and Analysis Plan 


As mentioned in Chapter 1, the evaluation team implemented a random assignment research 
design to estimate the impacts of access to the program. This design ensures that estimated 
effects can be attributed to access to the program and not to unmeasured differences in 
characteristics or external circumstances between individual students with access (treatment 
group) and without access (control group). 


This section describes the evaluation’s major components. It covers random assignment 
procedures, characteristics of the study sample, impact analysis methods, the implementation 
study approach, and data sources. 


3.3.1 Sample Enrollment and Random Assignment 


The evaluation team worked closely with Year Up’s national staff in designing study recruitment 
and enrollment procedures. The evaluation was designed to assign one eligible applicant to the 
control group for every two assigned to the treatment group. Local admissions teams thus had 
to recruit three eligible applicants for every two open program seats—a 50-percent increase 
over the number they normally would have had to attract to the program. The recruitment 
effort succeeded in generating a total sample of 2,544 young adults, with 1,669 treatment and 
875 control group members. 


With new cohorts starting the program twice a year, recruitment is an ongoing activity at Year 
Up. PACE enrolled two cohorts in each of the eight offices, with intake staggered at different 
times between January 2013 and August 2014 in different offices. Thus, although the sample 
built up over a two-year calendar period, it represents a full year’s worth of Year Up recruits 
nationally. 


To meet the study recruiting targets, Year Up received funding from the PACE evaluation for 
enhanced marketing and admissions screening. The program used some of the funds to hire an 
outside marketing agency, which tested recruitment messages, prepared materials, and worked 
with Year Up staff to implement approaches through a variety of online and traditional 
marketing channels. A separate report summarizes lessons from this experience (Fein 2016a). 


Year Up’s application process involves a series of steps. As described in Chapter 2, these steps 
include a mandatory information session, a formal application, drug screening and background 
checks, a “learning assessment” (group interview), one-on-one interviews, and a final staff 
meeting known as “decision forum” to discuss and select viable candidates. For PACE, local 
offices then took the additional step of randomly assigning eligible applicants to treatment and 
control groups. Just prior to random assignment, staff administered an informed consent form 
and two baseline data collection forms: 


e Informed consent form. Applicants who agreed to participate in the PACE study signed 
an informed consent form, which explained the study and their rights as a study 
participant. Those who did not consent to be in the study were not randomly assigned 
and were not eligible to participate in Year Up. 
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e Baseline data collection forms. Applicants who consented then completed two surveys. 
A Basic Information Form (BIF) collected standard demographic and economic 
information, and a Self-Administered Questionnaire (SAQ) inquired about a range of 
attitudes, beliefs, and psycho-social dispositions, as well as more-sensitive personal 
characteristics (e.g., financial security and criminal background). 


After applicants signed the consent form and completed the baseline data collection forms, 
staff used an online lottery tool developed for PACE to randomly assign them to the treatment 
or control group. Treatment group members had the opportunity to participate in Year Up (but 
were not required to enroll). Control group members could not participate in Year Up, but 
could receive other services available in the community. Program staff normally called study 
participants to inform them of their group assignments and provided control group members a 
list of alternative services in the community. 


National staff regularly assessed numbers at different stages of the admissions process (e.g., 
submitting an interest form, attending open house and formally applying, completing the 
learning assessment) to ensure local offices were on track to meeting the study’s sample 
targets. Local offices enforced a three-year embargo on Year Up services for control group 
members.” 


3.3.2 Characteristics of the Study Sample 


Exhibit 3-2 describes the study sample using a selection of characteristics measured through 
the BIF.”° The p-values in the last column test the hypothesis that treatment and control groups 
are similar for each characteristic. 


Results show that random assignment produced well-balanced treatment and control groups. 
Of 28 characteristics tested, only one shows a statistically significant difference: parent’s 
education (p=.064). Since, on average, three in 28 tests will show statistically significant 
differences due to chance at the 90-percent confidence level, this result demonstrates that the 
PACE study was highly successful in creating balanced groups. 


Baseline statistics also show that Year Up succeeded in reaching its target population of urban 
young adults. A majority of sample members were black (54 percent) or Hispanic (31 percent). 
Men (59 percent) outnumbered women (41 percent), though the latter figure is large for a 
tech-focused training program.’’ Most sample members (68 percent) were living with their 
parents, and few (nine percent) had children. Many had struggled in high school: 40 percent 
reported usual grades of C or below, and only 10 percent reported usually receiving A’s. About 
half had attended some college. Nearly two thirds (63 percent) were in families with annual 
incomes below $30,000. 


> Control group members who turned 25 during the embargo period (about half of the sample) were not eligible 


for Year Up when the embargo ended. 


© See Appendix Exhibit A-1 for definitions and details for these characteristics. 


7 For example, in 2014 women accounted for only 20 percent of recipients of Associate’s degrees in computer 


science nationally. See https://www.nsf.gov/statistics/2017/nsf17310/static/data/tab4-1.pdf. 
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Exhibit 3-2: Selected Characteristics of the Study Sample (N=2,544) 
Treatment Control p-Value of 
All Group Group Difference 
Characteristic Participants (T) (C) (T-C) 
Age (%) 811 
18 to 20 42.8 43.2 42.1 
21 to 24a 57.2 56.8 58.0 
Female (%) 41.0 41.0 41.0 .992 
Race-Ethnicity (%) 120 
Hispanic, any race 31.4 31.6 31.1 
Black, non-Hispanic 53.7 53.6 53.8 
White, non-Hispanic 5.5 5.1 6.2 
Another race, non-Hispanic 8.8 9.7 8.9 
Living Arrangements (%) 452 
Not living with spouse/partner or children 86.6 87.1 85.8 
Not living with spouse/partner, living with children 6.5 6.6 6.2 
Living with spouse/partner, not living with children 45 4.2 5.1 
Living with spouse/partner and children 2.4 2 2.9 
Living with Parents (%) 68.4 68.8 67.7 564 
At Least One Parent with Some College (%) 55.4 54.0 58.0 .064 
High School Grades (%) .249 
Mostly A’s 10.4 10.0 11.1 
Mostly B's 49.4 48.6 50.8 
Mostly C’s or below 40.3 41.4 38.1 
Educational Attainment (%) 191 
Less than a high school degree 0.6 0.7 0.5 
High school or equivalent 51.8 51.0 53.3 
Less than 1 year of college 22.1 22.3 21.5 
1+ years of college 22.5 22.9 21.8 
Associate’s degree or higher 3.1 3.1 2.9 
Received Vocational or Technical Certificate or 18.4 18.9 173 349 
Diploma (%) 
Career Knowledge Index (mean) 0.46 0.46 0.47 625 
Psycho-Social Indices (means)¢ 
Academic discipline 5.28 5.28 5.27 671 
Training commitment 5.52 5.52 5.50 232 
Academic self-confidence 5.05 5.04 5.07 226 
Emotional stability 5.33 5.33 5.32 .987 
Social support 3.35 3.34 3.36 .135 
Stress 2.20 2.21 2.18 315 
Depression 1.59 1.60 1.57 .103 
Family Income Last Year (%) 933 
Less than $15,000 37.1 37.3 36.7 
$15,000-$29,999 25.7 25.0 27.1 
$30,000 or more 37.2 37.7 36.2 
Family Income (Mean $) 27,021 27,287 26,528 443 
Continued 
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Treatment Control p-Value of 
All Group Group Difference 
Characteristic Participants (T) (C) (T-C) 
Public Assistance in Past 12 Months 
Received WIC or SNAP (%) 32.8 32.6 33.1 .756 
Received Public Assistance or Welfare (%) 6.6 6.3 7.3 352 
Financial Hardship in Past 12 Months (%) 29.7 29.4 30.3 078 
Current Work Hours (%) 490 
0 47.6 47.9 47.1 
1to 19 10.5 10.3 11.0 
20 to 34 26.7 27.4 25.3 
35 or more 15.2 145 16.6 
Expected Work Hours in Next Few Months (%) .866 
0 36.3 35.9 37.1 
1to 19 23.0 23.1 22.7 
20 to 34 31.1 31.5 30.3 
35 or more 9.6 9.4 9.9 
Life Challenges Index (mean)¢ 1.46 1.47 1.45 264 
Owns a Car (%) 28.8 28.7 28.9 .959 
Has Computer and Internet at Home (%) 84.9 84.1 86.5 111 
Ever Arrested (%) 16.2 16.6 15.5 502 
Sample Size 2,544 1,669 875 


SOURCE: Abt Associates calculations based on data from PACE BIF and SAQ. The means and percentages were calculated 
without weights. The p-values are based on t-tests (and for sets of categories F-tests) for differences between the two groups. 
SNAP is Supplemental Nutrition Assistance Program. WIC is Special Supplemental Nutrition Program for Women, Infants, and 
eal number of enrollees were accepted to Year Up just prior to their 25" birthdays and had reached age 25 by the time they 
completed the BIF. For simplicity, this table includes these sample members (less than one percent of the sample) in the 21-24 age 
Bot See Appendix Exhibit A-1 for operational definitions of indices and other variables in this table. 

Although Year Up screens for motivation, levels of training commitment in the Year Up sample 
were somewhat lower than for PACE overall (see Fein 2016b). About a quarter of Year Up 
applicants rated themselves as relatively highly motivated, compared with a third for the 
pooled PACE sample. The Year Up sample also was slightly less likely to report significant life 
challenges interfering with work, school, or family responsibilities (21 percent) than the overall 
PACE sample (27 percent). Young adults in the Year Up sample were more likely than all PACE 
participants to have an internet-equipped computer at home (85 percent, compared with 

72 percent) but less likely to own a car (29 compared with 56 percent)—perhaps because many 


Year Up participants came from central city neighborhoods (Abt Associates 2015). 
3.3.3. Impact Analysis 


The evaluation team conducted analyses to estimate the impact of Year Up on key outcomes 
and selected subgroups (see Chapter 6 for findings). Prior to estimating impacts, the team 
published an analysis plan specifying key hypotheses and outcome measures.~° The evaluation 
team subsequently assessed data quality, refined the plan, and publicly registered it on the 


8 See Abt Associates, Inc. (2015). 
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Open Science Framework website.”” The purpose of the analysis plan and registration was to 
document the evaluation team’s work and publicly commit to particular hypotheses and an 
estimation approach, consistent with ACF’s commitment to promote rigor, relevance, 
transparency, independence, and ethics in the conduct of evaluation.*° 


The impact evaluation was designed to estimate: 


e The impact of the offer to participate in Year Up—known as the “intent-to-treat” 
effect—not the impact of actual participation. That said, a very high percentage of 
treatment group members—96 percent—accepted the offer and began the program. 


e The impact of all components of Year Up combined, not the impact of specific program 
components. Because the offer included access to Year Up’s entire array of services, the 
random assignment design does not provide a rigorous basis for disentangling what 
individual program components contributed to impact. 


e The impact of Year Up in the context of other programs available locally. Both 
treatment and control group members could access other education, training, and 
support services in the community. The experimental design allows the evaluation to 
estimate the impacts of Year Up for individuals who may have had opportunities to 
participate in other programs and services. 


3.3.4 Hypothesis Testing 


An essential principle in the PACE analysis plan is to organize and discipline the number of 
statistical tests conducted. Like most social policy evaluations, the nine PACE studies target an 
array of different outcomes. If the evaluation did not adjust in some way for multiple 
hypothesis tests, at least some of a potentially large number of the tests in each study would 
reach conventional levels of statistical significance by chance—even if there was no true effect 
on these outcomes. This situation is known as the problem of “multiple comparisons.” To 
address the issue, the PACE evaluation established three categories of hypotheses: 
confirmatory, secondary, and exploratory. 


e Confirmatory hypotheses involve outcomes most critical to judging whether the 
program seems to be on track—that is, producing the results expected at a given follow- 
up duration. Given the relatively small sample sizes in the PACE studies, the research 
team generally limited such tests to one for each pertinent “main” outcome domain in 
the theory of change (see Exhibit 3-1). The sole pertinent domain for Year Up at this 
follow-up juncture is success in career-track employment. 


The confirmatory analysis estimated Year Up’s impact on the outcome: average 
quarterly earnings in the sixth and seventh quarters after random assignment. The 
evaluation team chose this period because these quarters immediately follow Year Up’s 
four-month period of post-program employment services. 


29 


See https://osf.io/bpm8u/. 


See https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy. 
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e Secondary hypotheses involve a set of additional indicators consistent with expected 
effects within the period covered by the study report. The number of tests is small to 
minimize the risk of false results (i.e., due to chance), which increases as the number of 
tests grows. 


Secondary analyses estimated Year Up’s impacts on annual earnings, indicators of 
career pathways employment, and self-assessed career progress roughly 18 months 
after random assignment. 


Each confirmatory and secondary hypothesis has an expected direction of change, an 
increase or decrease in the outcome. Accordingly, the associated statistical tests are 
one-tailed—testing only in the specified direction and ignoring possible effects in the 
other direction. 


e Exploratory hypotheses cover an additional set of possible effects whose direction and 
timing are less certain. Accordingly, the team applies two-tailed tests to these 
hypotheses. The Year Up analyses include tests of several kinds of exploratory 
hypotheses 


One category includes outcomes for which impacts may emerge later. For example, the 
program may improve students’ financial status and reduce financial strains, as 
reflected in measures of dependence on public assistance, financial hardship, access to 
health insurance, and other factors. However, these impacts are not expected to occur 
until after participants complete the program, obtain employment, and earn income for 
a longer period of time. 


e Asecond category includes outcomes for which the impacts could plausibly be positive 
or negative. For example, the program may reduce college enrollment and completion 
due to its emphasis on full-time employment after completion; alternatively, it may 
encourage college persistence after completion by offering college credit for Year Up 
courses and career guidance that covers college planning. Similarly, although the 
program might lead participants to postpone marriage and childbearing to focus on 
their careers, it also might increase marriage prospects by improving participants’ 
financial circumstances. Similarly, given uncertainty about how impact will vary across 
subgroups, the evaluation team also treats subgroup analyses as exploratory. 


e Athird category of exploratory hypotheses includes outcomes based on measures with 
uncertain validity. Most notably, standard measures of grit, savvy, and core self- 
evaluation may suffer from “reference bias,” which can arise when a program leads 
participants to see themselves as less skillful by raising their awareness of standards for 
psycho-social behavior.*’ Such bias will tend to cause estimated impacts to understate 
the true impacts of the program. 


37 For discussion of the challenges in measuring psycho-social skills, see Duckworth and Yeager (2015) and West 


et al. (2015). 
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3.3.5 Impact Estimation 


Following common practices, the evaluation team used a statistical technique to adjust 
comparisons of averages for the above outcomes between treatment and control group 
members. The adjustment helps to guard against any effects of chance differences arising 
during random assignment and can improve the precision of estimates.” The first step is to 
regress each outcome on a set of baseline variables for the control group. The next step is to 
apply the resulting regression model to calculate predicted values for the outcome for sample 
members in the two groups. The final steps are to calculate the average difference between 
actual and predicted values for each group and subtract the resulting average differences. 


More formally, the procedure applies the following equation: 


b= DAY) —Y(1-7)(¥-2) 


No F 


where © is the estimated impact of being in the treatment group (whether or not the person 


attended the program or used any of the offered services); Y is the outcome variable; Y isa 

prediction of Y based on baseline variables measured at random assignment; T is an indicator 
of treatment status (which is set equal to 1 if the individual is assigned to the treatment group 
and 0 if the individual is assigned to the control group); n, and n, are the respective sample 


sizes in the treatment and control groups; and the subscript i indexes individuals. Appendix A 
describes this procedure in greater detail. 


Once estimated, the evaluation team tested impacts for statistical significance at the 10-, 5-, 
and 1-percent levels. Tests of outcomes corresponding to confirmatory and secondary 
hypotheses were one-tailed, because expected effects were in a single direction. Exploratory 
analyses applied two-tailed tests, because the nature and direction of effects were less certain. 


The main data sources for the impact analysis were the baseline surveys (BIF and SAQ), the 18- 
month follow-up surveys of students, earnings data from the National Directory of New Hires 
(NDNH), and college records data from the National Student Clearinghouse (NSC). Section 3.4 
provides more details for these data sources. 


All analyses of follow-up survey data applied weights developed to adjust for differential 
nonresponse. Additional detail on these and other aspects of the analysis appears in 
Appendices B-F. 


3.3.6 Analysis Plan for the Implementation Study 
For the implementation study, the evaluation team conducted analyses of the design and 


delivery of Year Up services (in Chapter 4) and of differences in services received by treatment 
and control group members (in Chapter 5). Key analyses include the following: 


As explained in Appendix A, the approach is a variant on the traditional approach to regression-adjustment 


methods used in impact analyses. The latter typically involves linear regression of each outcome on an 
indicator of treatment status and a series of baseline variables. In this approach, the coefficient on the 
treatment indicator provides the regression-adjusted impact estimate. 
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Qualitative description of implementation. The description of Year Up’s design, theory 
of change, and context relies primarily on interviews with national and local office staff 
and review of program documents. In addition to site visits (described in Section 3.4), 
analysis draws on discussions in regular calls with program staff during the period of 
sample intake and random assignment. 


Quantitative analyses of participant flow. The evaluation team used administrative 
data from Year Up to document the proportions of participants having key program 
experiences and reaching major program milestones. These analyses detail outcomes 
for Year Up overall and for each of its eight offices. 


Fidelity. The team used multiple data sources to assess the degree to which Year Up’s 
actual implementation was consistent with its design and to document and understand 
changes in implementation over time. In addition to statistical analyses of 
administrative records for the entire program, analyses draw on interviews and 
observations from site visits to four local offices and the national office. 


Service differences. A key series of implementation study analyses document 
differences in services experiences between young adults who were randomly assigned 
to Year Up and the control group. In addition to establishing the intensity of services to 
treatment group members, these analyses establish the degree to which control group 
members were able to find comparable services in the community. The comparisons 
rely on service receipt each group reported in the 18-month follow-up survey and are 
supplemented with qualitative detail from in-depth interviews. 


Data Sources 


The evaluation draws on a wide range of data sources, including data collected directly from 
young adults; from the program’s staff and partners; and from records in several administrative 
data systems. Appendices to this report provide further details on most quantitative data 
sources. 


Data from surveys of young adults in the sample included: 


Baseline surveys. Prior to random assignment, all study participants completed two 
baseline forms. The BIF collected demographic and economic information. The SAQ 
measured a variety of attitudes, beliefs, psycho-social dispositions, and other more- 
sensitive personal characteristics such as criminal background and financial security. 


In-depth interviews. The evaluation team conducted in-depth interviews with random 
samples of participants in four local Year Up offices in 2014 (a total of 19 treatment and 
13 control group members). Interviews typically lasted 45-60 minutes and explored 
motives for applying to Year Up, experiences with Year Up and (for the control group) 
other services, and hopes and plans for the future. 


18-month follow-up survey. The survey covered education and training experiences, 
receipt of related supports, employment and self-assessed career progress, financial 
status, psycho-social outcomes, and family formation. The response rate was 78 percent 
for the treatment group and 73 percent for the control group. Appendix D provides 
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more detail on survey methods, imputation for item nonresponse, and adjustments for 
survey nonresponse. 


Information from Year Up’s staff and partners included: 


e Site visits. For the implementation study, the evaluation team conducted two rounds of 
site visits. In the first round in 2012-2013, the team visited all eight local offices and 
focused on admissions and random assignment procedures. In a second round in 2014, 
they visited four of the Year Up core program’s eight offices (Bay Area, Boston, Chicago, 
and Washington, DC) to assess implementation. The team selected these offices to 
represent different regions of the country and different local operating environments 
(e.g., local populations, labor markets). During the visits, researchers interviewed a wide 
range of staff, at Year Up and at its college and employer partners; observed key 
program activities; and gathered program documents. In 2014, the team also visited 
Year Up’s national office to interview program leaders. 


e Staff survey. The evaluation team administered an online survey to Year Up staff in local 
offices covering attitudes about the program, current practices, and other topics. About 
half of Year Up’s regular full-time local staff members (153 respondents) completed the 
survey. 


Administrative data include records from several automated data systems: 


e Year Up’s management information system. Data in a December 2016 extract from 
Year Up’s primary management information system—a Salesforce® database—covered 
key events arising in connection with admissions, contract infractions and stipend 
receipt, program retention, employer payments, and employment outcomes as of Year 
Up’s four-month post-graduation follow-up. A number of indicators in Year Up’s FM- 
RADIO performance management tool (see Chapter 1) also draw on these data. 


e National Directory of New Hires (NDNH). Maintained by the federal Office of Child 
Support Enforcement, the NDNH includes an individual’s quarterly earnings across 
multiple employers, excluding earnings from self-employment and “off the books” jobs. 
Data from an April 2018 extract cover a period extending from two quarters before 
random assignment to three years (12 quarters) after random assignment for all sample 
members. 


e National Student Clearinghouse (NSC). Maintained as an enrollment verification service 
for purposes of administering student loans, the NSC provides very high coverage of 
enrollment at public two- and four-year colleges (more than 96 percent) and private 
nonprofit (96 percent) colleges. It also provides good coverage of for-profit four-year 
colleges (81 percent), but coverage is relatively low for nonprofit (42 percent) and for- 
profit (27 percent) two-year colleges. Data from an April 2017 extract cover enrollment 
outcomes over nearly three years (11 quarters) after random assignment. 
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4. Implementation Analysis 


Chapter 2 described the wide array of services that local Year Up offices must develop and 
integrate to deliver the program as envisioned. This chapter assesses the offices’ approach to 
implementing and operating these services during the PACE study period (2013-2014). It draws 
on interviews with national and local staff and with the program’s college and employer 
partners, as well as quantitative analysis of program data (described in Chapter 3). A following 
chapter (Chapter 5) compares services received by treatment and control group members, 
analyzing data from the 18-month follow-up survey and in-depth interviews with young adults 
in each group. 


The chapter begins with an assessment of responses to several cross-cutting organizational 
challenges. Section 4.2 summarizes efforts to meet ambitious recruitment goals. The next three 
sections discuss the implementation of Year Up’s basic services— education and training (4.3), 
“high support, high expectations” strategies (4.4), and internships (4.5). Section 4.6 examines 
career transition services. A final section (4.7) summarizes the main findings. 


4.1 Organizational Aspects 


This section assesses Year Up’s approach to several cross-cutting requirements for consistently 
delivering complex and resource-intensive services in multiple offices. One response to this 
challenge has been to promote a clear set of shared norms, values, and practices for staff and 
program partners. Another requirement involved establishing a sustainable financial model. A 
third need was to effectively organize and manage staff across eight local offices and find the 
right balance between national and local levels of authority. 


4.1.1 Culture 


In interviews, many stakeholders emphasized the strong role that culture and values play in the 
Year Up program. They cited many ways the organization aims to foster active reflection on, 
and shared commitment to, its mission and core values. For example, staff at all levels reported 
spending a great deal of time discussing goals and values and assessing and revising processes 
to promote desired outcomes. Year Up’s core values apply to staff as well as students. They 
include “respect and value others,” “build trust and be honest,” “engage and embrace 
diversity,” “be accountable,” “work hard and have fun,” and “strive to learn.” 


The organization used several strategies to foster internal alignment on its mission and values. 
For example, in hiring staff, leaders put much weight on applicants’ passion for helping young 
adults. According to one local staff member, the message from national leaders is, “I don’t care 
how much you know. | want to know how much you care.” National leaders also promote 
extensive discussion and communication about mission and values. Having all staff serve as 
advisors to multiple students over the course of successive one-year cycles is intended to help 
staff keep sight of the program’s desired end results. 


Many informants noted that Year Up’s cultural ethos played a strong role in motivating and 
aligning staff. Two emphases are particularly salient—a pervasive use of language and practices 
from the business world and a strong emphasis on continuous learning. 
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e Year Up’s national and local leaders created an organizational culture that emphasized 
concepts and tools from the business world. 


Interviews, observation, and documents gathered during site visits revealed many ways that 
Year Up has woven business concepts and practices into its program’s fabric. This aspect of 
program culture is intentional and rooted in the fact that many national and local leaders have 
degrees or experience in business or management. 


The evaluation team observed many examples of this business ethos during site visits to 
program offices. It is embodied in standards for professional conduct and dress, student 
contracts, the physical look of local offices, teaching methods that encourage initiative, and 
training in business tools and practices. Instructors assign “deliverables” rather than homework, 
and in some offices refer to students as “junior colleagues.” Staff members often refer to Year 
Up as “the company.” 


Local staff noted that fostering a businesslike, corporate environment helps to create strong 
socializing influences on participants. As one interviewee put it: 


Year Up provides a very consistent structure... [I]t’s not loose. You know what to 
expect: [the hours are] 8:30-3:30, your stipend is this, and [there is] a particular 
routine. Some of our young people are coming from environments where there 
was never a routine: you could show up at high school or not show up, do 
whatever you want to do. But Year Up creates a culture similar to the culture we 
are trying to have them assimilate in. 


Year Up’s national leaders described strong relationships with the social entrepreneur and 
venture philanthropy communities. They use tools adapted from the private sector in 
fundraising and engaging corporate partners. For example, in fundraising Year Up uses 
prospectuses modeled on investment capital solicitations to encourage donations based on 
anticipated social and financial returns. 


e Year Up applied its core value “strive to learn” to both students and staff. 


Interviews, classroom observations, and curriculum documents revealed many strategies Year 
Up uses to encourage students to take responsibility and actively engage in learning. As 
described in Section 4.3 below, instructional methods are project-based and emphasize critical 
thinking, problem solving, and individual accountability. Instructors teach students how to 
solicit and provide feedback from other students and staff and encourage them to ask 
questions. 


Staff informants also noted that the organization encourages staff to apply the value “strive to 
learn” in ongoing assessment and continuous improvement of program services. Formal 
mechanisms include annual staff retreats to review and revise approaches, “no-blame look- 
back” assessments in local offices between each six-month L&D and internship session, cross- 
office calls to share experience and best practices, electronic platforms for sharing ideas and 
examples of promising service approaches, and support for staff development and training. 
Year Up encourages feedback from students to staff, between staff, and to and from employers 
and other program partners. Local staff frequently pilot new approaches to curricula and 
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services—examples include group projects spanning multiple courses and a new course 
emphasizing basic math skills. 


The organization has invested heavily in its management information systems (particularly its 
main Salesforce® database), and performance statistics derived from these systems (e.g., the 
FM-RADIO metrics) figure heavily into pilot assessments as well as more global improvement 
initiatives. 


4.1.2. Finances 


Year Up’s average cost was $28,290 per student in 2013-2014.*° Local staff and student 
stipends (averaging $6,614™ per student) accounted for a majority of that total (46 and 23 
percent, respectively), and other local program expenses generated 20 percent of costs (Exhibit 
Ast? 


e Year Up covered these costs with substantial revenue from employer internship payments 
and philanthropy and very little reliance on public funds. 


Nearly 60 percent of local operating costs were financed by employer payments for interns, as 
shown in the right-hand panel of Exhibit 4-1. Foundation grants covered 22 percent, and private 
companies and individuals supported an additional 17 percent through charitable donations. 


Public funds accounted for less than two percent of support for Year Up services. Leaders 
explained that they seek to minimize reliance on governmental grants in order to reduce 
exposure to unpredictable fluctuations in funding, maximize flexibility in use of funds, and 
minimize sometimes onerous reporting requirements. 


Experience has proven this financial model to be sustainable at the program’s current scale. As 
discussed later in this chapter, Year Up has identified and is testing revisions to the financial 
model to support further up-scaling. 


e Each local Year Up office is responsible for raising funds to cover its operating costs— 
primarily including staff salaries, student stipends, rent and other administrative 
expenses. 


In Year Up’s FM-RADIO framework (described in Section 2.4 of Chapter 2), annual financial 
management (“FM”) and development (“D”) goals are key performance metrics. The national 


3 This analysis uses Year Up aggregate financial data for 2013-2014 to approximate costs and revenue during 


the period of services to the PACE sample. The statistics reflect financials for some cohorts enrolling before 
and after PACE cohorts (as explained in Chapter 3, local offices enrolled PACE cohorts on a staggered basis 
during this period). The numbers accurately represent average costs and revenue over the two calendar years 


that the PACE sample received services. 


** As discussed in Chapter 6, the average stipend was slightly higher—$7,172—for the subset of 1,560 young 


adults assigned to the treatment group for PACE. 


2 The $28,290 figure includes the direct costs of training, all of which Year Up staff provide. Most college 


partners also charge tuition in exchange for granting credits for Year Up—provided training. These costs are 
financed partly by Pell and other student aid programs, partly by college contributions, and partly from Year 
Up revenues. Separate data on the amounts (included in the “program, fundraising, and administrative costs” 
category) were not available. 
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team provides substantial support by leveraging relationships with larger foundations and 
companies to set up opportunities for local development teams. The system operates like a 
bank, with surpluses and debts carrying over across years and loans provided from national 
accounts as necessary. 


Exhibit 4-1: Program Costs and Revenue for 2013-2014 


100% 
Public & misc. contributions ($633) [LESTE 


National office costs ($3,007) Corporate contributions ($1,300) 


Individual contributions ($3,531) 


Program, fundraising & 
administration costs ($5,610) 


Foundation grants ($6,115) 


Student stipends ($6,614) 


Internship revenue ($16,907) 


Personnel costs ($13,055) 


Components of Average Average Revenue per 
Cost per Participant Participant by Source 
(Total=$28,290) (Total=$28,487) 


SOURCE: Financial summary from Year Up for all participants served in 2013-2014 (n=3,539). 


In interviews, local development leads identified creative fundraising strategies and had 
perceptive insights into local fundraising environments. Many engaged students as fully as 
possible, because, as one staff member put it, “They tell our story better than we ever could.” 
Development teams encouraged potential sponsors to visit Year Up offices and participate in 
program activities. One informant said that they liked to have visitors work with students on 
Year Up “interactives” involving creative problem-solving tasks. She said, “It gets the foundation 
folks in particular. ... It’s fun for them. Makes them do a little work, shakes things up.” 


Year Up’s message reportedly resonates with many donors because they recognize the 
difference support has made in their lives, and because Year Up offers a chance to help solve 
wider societal problems. Teams modulate the pitch depending on the sponsor—one 
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development lead cited as an example putting more emphasis on benefits for business and the 
wider economy in discussions with conservative donors. 


Most offices coordinate development activity with efforts to develop (in Year Up parlance 
“sell’) internship seats and engage mentors and other volunteers in the program. Staff stressed 
the fluid and developmental character of relationships across these functions. For example, a 
manager at a local firm might start as a mentor and then become a donor or internship 
sponsor. 


4.1.3. Organization and Staffing 


Delivering Year Up’s comprehensive services in varied localities poses a series of staffing and 
organization challenges. 


e One challenge was finding the right balance between national and local office roles. 


The PACE period followed several years of rapid growth in the number and size of local Year Up 
offices. From 2008-2013, the program added five offices and roughly tripled the number of 
young adults served annually (see Fein 2016a). 


In interviews, Year Up’s leaders noted that growth led to greater diversity in local practices and 
concerns about consistency in the quality of services. They sensed a need for increased national 
control but did not want to discourage positive adaptations to local conditions. 


For their part, local staff valued their autonomy and believed they were best equipped to 
respond to local needs. They noted a tendency at times for national office staff to over- 
troubleshoot without fully understanding local circumstances, and to launch new initiatives 
without fully considering the burdens for local staff. 


In response to feedback from local offices, Year Up’s leaders emphasized to national staff that 
their main job was to “perfect services to sites.” To foster understanding of local programs, 
nearly all national staff must work in a local office for six months sometime during their first 
two years of employment. For national program staff, perfecting services to sites includes 
supporting cross-site dialogue, dissemination, and adoption of best practices. For most local 
staff roles, national staff organize an annual retreat and monthly cross-site teleconferences to 
discuss common challenges and share insights and strategies. 


Another response to identifying the right level of involvement by national staff involved use of a 
common set of performance measures—FM-RADIO (see Chapter 2). This tool helped national 
leaders to discern productive from problematic deviations in local practices. As one national 
leader put it, “Hitting the FM-RADIO [target] is the price of freedom: If you hit it, | will clear 
almost everything | can, whether it is experimenting with something outside the box or hiring 
more people.” 


To foster consistency and scalability, the national team began to standardize and centrally 
manage key program services. One approach to standardization involved creating “playbooks” 
of guidelines for practices—for example, an early admissions playbook drew on lessons learned 
in expanding recruitment for PACE. Three new National Site Directors assumed responsibility 
for managing local offices on a regional basis. The national Corporate Engagement team 
assumed supervisory responsibility for local staff members working on employer partnership 
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development. The national team also took initial steps to centralize curriculum planning and 
development. 


e Asecond organizational challenge was knitting together the program’s varied 
components into a coherent whole. 


National leaders observed that increases in the number of staff and specialization in functions 
and services could lead to a “hand-off mentality”—that is, a tendency of staff to focus on their 
specific roles and not consider young adults’ needs in a holistic manner. Rapid hiring reportedly 
exacerbated the problem by generating a substantial influx of young and inexperienced staff. 


Staff in different roles sometimes saw program goals differently. Leaders cited the example of 
Year Up’s admissions (closely linked to social services) and Corporate Engagement teams. The 
former tended to focus on the program’s mission of serving youth who need help, whereas 
internship sales teams focused on convincing employers that Year Up was a dependable source 
of entry-level talent. In recruitment, the former were inclined to take on more needy 
applicants, while the latter pressed for more stringent screening. 


Leaders’ approach these tensions involved promoting shared values and accountability for FM- 
RADIO outcomes. They used statistical analyses of the relationships between performance 
outcomes to encourage alignment on approaches. Such analyses helped to convince staff of the 
strong interdependencies between student characteristics, retention, and internship sales, for 
example. 


Year Up’s advising system also helps to promote integration. As noted earlier, every national 
and local staff member advises four to eight students throughout the program and participates 
in Friday Feedback and other activities as a member of the learning community. Many staff 
praised this approach for its effectiveness in widely familiarizing staff with all aspects of the 
program and fostering responsibility for helping advisees negotiate challenges and arrive at 
positive outcomes. 


In addition to serving to introduce students to Year Up, a robust orientation program for 
incoming students also functions as an integrative strategy for staff. Every six months, most 
local staff members participate in a week-long orientation session for incoming students. The 
intensive array of presentations and interactive activities reportedly has strong positive effects 
in engaging new staff in Year Up’s culture and processes and in reinforcing knowledge and 
connections among existing staff. 


e Athird challenge was keeping Year Up well staffed during a period of increased 
specialization, rapid growth, and heightened expectations. 


During in-person interviews and in the PACE staff survey, many staff identified burnout and 
turnover as challenges for Year Up. The program’s weekly operations and bi-annual recruitment 
and internship targets imposed exacting demands on local teams. Year Up’s strong focus on 
mission encouraged staff to go above and beyond assigned responsibilities and work hours 
when students need extra help. Its emphasis on continuous improvement reportedly leads 
many staff to devote extra effort to devising and testing new approaches. As one local staff 
member summarized the situation: 
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There is a point where you roll your eyes [about the ethos of “never say no”]. ... 
But it’s the mission of everyone here. People work at Year Up because they care. 
It’s hard work to be in the trenches. You see a lot. It’s intense. We train people to 
have tough skin. 


In response to these pressures, national leaders described a number of measures to reduce job 
stress. These steps included a generous benefits package, a wellness initiative, and efforts to 
strengthen career paths within the organization. National leaders also sought to minimize 
burdens on local teams from new projects launched by national staff. They also sought to 
improve efficiency and reduce workloads. For example, an initiative to place larger numbers of 
interns at fewer employers was motivated in part by the greater efficiency and reduced 
workloads such a model implies for internship sales staff. 


Interviews also highlighted the particular challenges in staffing some specialties. The most often 
mentioned recruitment challenge was IT instructors, who typically command substantially 
higher salaries in private industry and college settings and may have little experience with 
nontraditional teaching methods. Good fundraisers also were difficult to recruit and retain—a 
challenge for other nonprofit workforce programs as well as Year Up. 


4.1.4 Recruiting Participants 


During PACE, a considerable additional challenge for local offices was meeting stepped-up 
recruitment targets to generate a control group. As discussed in Chapter 3, the study required 
recruiting 50% more qualified candidates than needed to fill program slots. The effort was 
successful and, in addition to meeting the targets, generated many useful lessons on scaling 
recruitment (Fein 2016a). 


e Alllocal offices were able to increase recruitment by 50 percent across two cycles to 
generate the PACE control group. 


All offices hit their target numbers for PACE. The overall acceptance rate and average number 
of assessed risks were relatively constant for cohorts recruited before, during, and after PACE 
(see Fein 2016a). The implication is that the program maintained its rigorous screening 
practices even while increasing the scale of recruitment efforts. Year Up accepted 16 percent of 
applicants during PACE (see Exhibit 4-2). As noted in Chapter 1, stringent screening is typical for 
sectoral workforce programs that, like Year Up, strive to meet the needs of both participants 
and employers (e.g., Hendra et al. 2016). 


Meeting stepped-up recruitment goals was challenging. Although staff and community partners 
mostly understood the value of random assignment and that it had little bearing on the number 
of available program seats, many found it difficult to deny admission to control group members 
after intensive efforts to engage them. In interviews, admissions staff also cited the difficulty of 
accurately gauging interest in IT and other Year Up occupational specialties from essays and 
interviews. Some applicants reportedly overstated their interest in IT to enhance their chances 
of acceptance and later exhibited low motivation and performance in technical courses. 
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Exhibit 4-2: Selected Implementation Measures by Office 
All 
Outcome Offices Office A  OfficeB  OfficeC  OfficeD  OfficeE OfficeF OfficeG  OfficeH 
Percentage of Applicants Accepted 16.2 11.1 28.7 12.2 47.8 16.7 12.0 14.5 12.7 
Percentage of All Treatment Group Members: 

Assessed as: 

Low risk 28.1 20.9 21.6 17.1 47.9 27.3 27.4 25.2 33.1 

Intermediate risk 42.6 34.2 37.9 34.8 32.2 49,7 48.9 61.4 50.6 

High risk 29.3 44,9 40.5 48.1 19.9 23.0 23.7 13.4 16.3 

Total 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 
Who started L&D phase activities 96.0 95.7 94.7 99.4 98.0 91.0 96.0 97.5 96.3 
Who completed the L&D phase 81.5 73.8 79.1 90.6 82.0 75.8 82.4 83.4 87.7 
Who started internships 80.7 72.0 79.1 89.9 79.5 75.8 82.0 81.6 87.7 
Who enrolled in college during Quarters 0-3 65.2 78.7 73.4 96.2 88.1 43.0 11.9 57.7 93.2 
Sample size 1,669 164 321 159 244 178 278 163 162 

[Percentage of Those Starting 6D 

Who completed the L&D phase 85.0 77.1 83.6 91.1 83.7 83.3 85.8 85.5 91.0 
Who started internships 84.1 75.2 83.6 90.5 81.2 83.3 85.4 83.6 91.0 
Received contract infractions during Year Up 

0 infractions 4.4 13 7.6 11.4 17 1.2 11 6.3 5.8 

1-5 infractions 30.6 21.7 39.5 33.5 18.4 22.8 30.0 49,7 27.6 

6-9 infractions 20.3 19.7 19.4 22.8 19.7 18.5 17.6 22.6 25.0 

10+ infractions 44.7 57.3 33.6 32.3 60.3 57.4 51.3 21.4 41.7 

Total 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 
Sample size 1,602 157 304 158 239 162 267 159 156 


SOURCE: PACE calculations based on a December 2016 extract of participant records in Year Up’s Salesforce® management information system. 

NOTES: Offices ordered from smallest to largest impact on the confirmatory earnings outcome (see Exhibit 6-11-in Chapter 6). The “low, intermediate, and high risk” categories refer to 
participants who scored 0-1, 2-3, or 4 or more, respectively, on a Year Up “sum of risks” measure. To derive this measure, Year Up sums ratings across 14 factors assessed for each 
applicant using a 0-4 severity scale (with 0 indicating no risk). Risk factors include issues like transportation, housing, childcare, substance abuse, exposure to domestic violence, and 
mental health. 
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e Most offices met the PACE recruitment targets without digging deeper into their existing 
applicant pools and accepting “higher-risk” applicants. 


As Exhibit 4-2 shows, six of the eight offices accepted fewer than 20 percent of applicants, and 
only one accepted a substantially higher fraction (Office D, at nearly 50 percent). In itself, a high 
acceptance rate does not necessarily imply less-stringent screening: initial applicant pools can 
vary in the proportion of suitable young adults they contain. For example, although Office D 
accepted nearly half of applicants, the percentage of admissions classified as “high risk” 

(20 percent) was lower than in the overall sample (29 percent). °° 


Conversely, two offices with very low acceptance rates (Offices A and C) had higher proportions 
of high-risk admissions (45 and 48 percent, respectively). These offices likely were recruiting 
from more disadvantaged local populations. 


Year Up leaders noted that some offices have greater capacity to support high-risk students 
than others. Statistics in Exhibit 4-2 hint at such variation, showing, for example that the office 
with the highest percentage of high-risk students (C) also had the highest rate of L&D 
completion. As one leader put it: “[The optimal] balance is probably different in every site... 
there’s a constant debate between who we admit, how we do it, and how we execute.” 


4.2 Education and Training during the Learning and Development Phase 


This section assesses the customized training in business communications, technical, and 
professional skills that Year Up provides during the 21-week L&D phase. It starts with a brief 
look at statistics on L&D participation and then turns to curriculum development and 
instructional practices. 


e High percentages of treatment group members began and completed the 21-week L&D 
phase. 


Of sample members assigned to the treatment group nearly all (96 percent) enrolled in the 
program and began classes. A very high fraction of these enrollees (85 percent) completed the 
L&D phase (see bottom panel of Exhibit 4-2). The latter figure exactly meets the 85-percent 
retention standard for enrollees set by Year Up’s national office (the “R” in FM-RADIO). As 
shown in Exhibit 4-2, retention reached 91 percent in two offices and fell below 80 percent in 
one office. 


An implication is that impacts measured in PACE represent the effects of a program that 
achieved the full dose of instruction for a very high proportion of young adults offered services. 


3° The low, intermediate, and high risk categories refer to participants who scored 0-1, 2-3, or 4 or more, 


respectively, on a Year Up “sum of risks” measure. To derive this measure, Year Up sums ratings across 14 
factors assessed for each applicant using a 0-4 severity scale (with 0 indicating no risk)—possible values 
accordingly ranged from 0 to 56. Risk factors include issues like transportation, housing, childcare, substance 
abuse, exposure to domestic violence, and mental health. 
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e Most Year Up offices were agile in revising curricula in response to changing employer 
needs. 


In general, local office staff cited active efforts to solicit and act on feedback from employers on 
needed skills. Mechanisms included online surveys eliciting supervisors’ assessments of each 
intern at the middle and end of their internship, ongoing monitoring of interns by program 
staff, and substantial contact with employers. 


One office’s approach exemplifies the extent to which local offices could go in soliciting 
employer feedback and developing new curricula. Largely in response to employer-identified 
needs, this office expanded from two to four technical tracks (IT helpdesk, quality assurance, 
project management, and cybersecurity). Students took foundational technical courses 
together and then divided into more specialized courses, structured to shift from cohort to 
cohort in accordance with changing employer needs. The site also worked with a leading 
cybersecurity software developer to develop a customized curriculum establishing new 
certification points for entry-level positions. 


Substantial curriculum change—whether in existing or new courses—required review and 
approval by college partners for the course to offer credit. In many instances, revisions to 
existing courses received favorable, often pro forma review. 


In other situations, staff opted to pilot supplemental training in ad hoc formats on a non-credit 
basis. One office piloted an elective in customer service that subsequently was integrated in the 
main curriculum. Another office piloted an elective in business math that was discontinued 
when it did not appear to address employer needs. Another implemented a five-week quality 
assurance boot camp to a subset of students at the end of L&D. Although this training did not 
generate college credits, instructors believed it was useful in the job market. 


The pace of curriculum revision was slower in some offices than in others. Staff explained that 
in some fields, the skills that employer partners needed changed more slowly. They said that 
signals from employers often did not add up to a strong enough pattern to justify change. But 
sometimes staff said that their office was not as nimble as it should be. One expressed 
frustration that it often took a year or more to develop and launch a new course. 


A general challenge cited by a number of instructors was that schedules already packed with 
teaching, learning community (e.g., advising), and administrative duties left little time for the 
intensive study and discussion required for curriculum development. They typically focused on 
this task during intersession periods in February and August— periods described as “really 
intense” because staff also had to train new instructors and handle other duties during this 
time. 


e With growth in the number and size of local offices, the national office began to increase 
its involvement in curriculum monitoring and alignment. 


As noted in Section 4.1 and Fein (2016a), the number and size of local offices were increasing 
rapidly in the years preceding PACE. As Year Up grew, it became increasingly difficult for the 
national team to keep up with local curriculum developments and practices. During the PACE 
period, national academic leads took steps to monitor and guide these developments more 
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closely. They also introduced a variety of measures aimed at promoting alignment on best 
practices. 


For example, national office leaders instituted a variety of cross-office meetings involving 
instructional staff. Formats included a monthly teleconference and biannual in-person meetings 
with local office academic directors. Leaders also organized virtual “mini-conferences,” in which 
instructors took turns leading short trainings and sharing best practices with other instructors. 
Instructors also shared curricula and materials via the online instructional platform Schoology. 


Towards the end of the PACE period, national leaders decided that a more centralized approach 
to curriculum development was needed. The decision was based on the increasing difficulty of 
monitoring and ensuring quality of proliferating curricula, the growing burden for already 
stretched local instructors, and the mounting inefficiency in duplicating efforts across offices. 
Work on this centralized capacity got underway in 2015-2016. 


e Year Up courses engage students in active discussion and emphasize group work and 
project-based learning. 


Of the 51 instructors responding to the PACE staff survey, very high fractions said that their 
Year Up teaching often or nearly always involved project-based learning (86 percent) and 
working in groups (96 percent). Only 31 percent described their classes as often or nearly 
always lecture-based. Supporting such assessments, more treatment than control group 
students cited promising instructional approaches in the 18-month follow-up survey (see 
Chapter 5). 


Classes observed during site visits were active, engaging and lively. In on-site interviews, 
instructors emphasized that hands-on and interactive learning is especially critical in working 
with the many Year Up students who have not had good prior experiences with school. 


In interviews, instructors cited a variety of approaches to active learning. As one instructor said, 
“| like to bring out a lot of interaction, peer to peer, a lot of assignments in work and reading. 
We use video, TED Talks, a lot of hands-on equipment itself, critical thinking, problem solving.” 


Group projects often formed the foundation of the curricula, particularly in technical training 
courses (IT and financial operations). One example is a “Finopoly” project (based on the board 
game Monopoly) one local office used in a seven-week Financial Management course. The 
game simulated accounting processes over a corporation’s financial cycle. Students split into 
small groups acting the part of companies in different industries. Based on the group’s cash 
holdings and financial transactions represented by rolls of the dice, students recorded changes 
and create financial statements. At the conclusion of the “fiscal year,” each group prepared an 
annual report, pitched “investors” to take the corporation public, and purchased shares in 
other corporations. 


Another office ran a cross-site project involving student teams at two locations. Each team had 
to propose a business idea, develop the details, and present its proposal to “investors.” Project 
work required mastering Agile project management skills and drew on skills taught in the 
office’s technical and Pro Skills courses—including skills in group facilitation and teamwork, 
conflict resolution, and understanding different learning styles. Staff noted that, in addition to 
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these skills, the cross-office aspect fostered expertise in the technology and etiquette of virtual 
communication central to many business environments. 


e Instructors take varying approaches to teaching students at different levels of ability. 


Year Up admits many students assessed at enrollment to have lower basic academic skills (see 
Exhibit 3-2 in Chapter 3). Strategies for teaching lower-skilled students varied. One instructor 
paired lower-skilled students and more-advanced students on group projects to encourage 
peer support. Another instructor explained how learning difficulties often required slowing the 
pace: “It’s really important that these students get these particular skill sets that we’ve 
identified. [Our approach is:] ‘If we’re not getting it, we’re going to spend more time on it. 


wn 


Such an approach could make it difficult to keep courses on schedule and to keep more 
advanced students engaged. One instructor explained his approach to meeting needs at varying 
ability levels: 


We’re better now than we were in the past at differentiating our instruction. This 
student is lower, this upper, I’ll give them this to do. Additional readings, they can 
go further in a Schoology class. ... We can differentiate with the [learning 
management system] online. They can have different units. 


Year Up instructors track students’ classroom performance and written assignments and exams 
closely. As described in Chapter 2, in addition to graded classroom assignments, students 
receive feedback from 7- and 14-week assessments. The assessments help instructors and 
students identify additional steps to improve academic outcomes. 


e Agreements with partner colleges varied in the college credits students could receive 
during Year Up and in the qualifications applied to Year Up instructors. 


Each Year Up office worked with its local college partner to maximize the number of awarded 
credits that could be transferred to other colleges. One Year Up office aligned its Business 
Communications course with its partner college’s English composition course, because credits 
for the latter were transferable to four-year colleges. Staff in several offices noted that limited 
transferability was often a problem with credits earned during Year Up. For example, at one 
office, only 14 of the 24 total credits from the community college partner transferred to the 
state university system. In another office that partnered with a four-year college, the latter’s 
credits did not easily transfer to two-year schools. 


College standards had an important bearing on instructor hiring. College partners typically 
treated Year Up instructors as adjunct faculty and often required a master’s degree in the 
discipline taught. Year Up staff at several offices reported that it was especially challenging to 
recruit IT instructors with the requisite credentials who were willing to accept lower salaries 
than they could command in private industry. Some college partners were more flexible and 
counted industry experience and certification in lieu of a master’s degree. 


e Differing arrangements with partner colleges led to substantial variation across offices in 
proportions of Year Up students formally enrolled in college during the program. 


According to college records in the National Student Clearinghouse, 65 percent of treatment 
group members were registered as enrolled college students at some point during their year in 
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the Year Up program (see Exhibit 4-2). The percentage enrolled varied from 12 to 96 percent 
across the eight local offices. 


Most Year Up offices helped students navigate college enrollment processes in college after 
admission to Year Up. The office with 96 percent enrolled in college provided such services 
during the admissions process. It took this approach at least in part so that control group 
members would have their paperwork ready and not miss the chance to enroll in college due to 
time spent applying to Year Up. 


e /T instructors believed that industry certification was valuable but generally did not favor 
spending time preparing students for certification at the expense of other program 
activities. 


Preparing students for certification and licensing exams in IT and other training-related fields 
was not a standard component of Year Up’s model. Instructors cited several reasons for not 
making this outcome a high-priority goal. Some felt that the short time students were in Year 
Up was best used in equipping students with strong basic technical skills and setting them up to 
pursue credentials on their own or with help from employers after the program. Other staff 
members believed that certification was not critical to employment immediately following the 
program, since employers got to know students’ skills through internships and generally trusted 
the Year Up brand. 


Although not a central component of the model, staff in most offices recognized that 
certification was very important at some firms and for longer-term advancement. One 
interviewee estimated that CompTIA A+ certification was worth an additional $5-$10 more per 
hour to a Year Up graduate. Another noted that A+ and Network+ certification were especially 
helpful in federal government jobs. Accordingly, many offices offered ad hoc workshops, short 
courses, and financial support with examination preparation materials and fees. 


4.3. Implementing Year Up’s “High Support, High Expectations” Strategies 


As described in Chapter 2, Year Up provides a comprehensive array of supports. This section 
assesses the level of implementation and varying approaches to these components in local 
offices. Analysis focuses on five of the program’s most salient supports: advising, support 
services, financial stipends, contract enforcement, and feedback. 


e Year Up’s advising system created a strong layer of supports and connections between 
students and staff. 


The requirement that nearly all local and national staff serve as student advisors added a 
substantial time commitment to their regular duties. Although time consuming, most staff 
interviewed emphasized the requirement’s value in familiarizing and holistically engaging them 
in the program. They also cited many examples of how regular interaction with advisors 
increased the likelihood of students sharing and getting timely support with problems. 


37 Asa result, control group members received some support applying for college that they would not have 


received absent Year Up. The effects of this support appear to have been modest, however, as fewer than half 
(42 percent) of control group members in this location went on to enroll in college. 
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Although general guidelines require advisors to maintain weekly contact with advisees during 
L&D (and visit periodically during internships), advisors have much flexibility to choose formats 
and activities. One advisor’s account suggests the flexibility this approach engenders: 


Last week [my advisees and I] took a walk over to [the local park], and we talked 
about a lot of things bothering them, their goals for this week, the Kardashians. 
Whatever they needed to talk about. | like to do things, introduce them to new 
things. But at the same time, if we just need a chat session, that’s what we’Il do. | 
know some people are a lot more structured in this advising time. For me, it’s a 
nice time to just be together and talk about what’s going well and what’s not, 
and be that support for them. 


e Year Up’s social workers identified housing and mental health as among the more difficult 
challenges to address. 


In in-depth interviews during site visits, Student Services staff emphasized the difficulty of 
finding suitable housing for young adults (and related challenges, such as homelessness and 
problems at home) and identifying and addressing participants’ mental health issues. In most 
Year Up cities, housing costs are high and supports are limited for young adults. One staff 
member said that public agencies see young adults’ needs as less “dire ... that they can find a 
friend’s couch.” But the reality is often different. As one student described it: 


That was my biggest challenge—just finding somewhere to crash every night... 
that was it. [YJou’re not gonna pay first, last, and security on a Year Up salary 
while [paying to have] the lights come on every day and still eat and live. It was 
just a situation | couldn’t find a way to make sense of. 


Although there appeared to be relatively few students with severe mental health problems, 
social workers stressed that difficulties in detecting and addressing such issues also put them 
near the top of their list of challenging problems. 


e Providing social services required successfully negotiating a number of organizational 
challenges. 


Another often-cited challenge for Student Services staff was meshing with other staff. They 
noted that it could be difficult to clearly define and communicate their role in an environment 
with many other committed and involved adults. As one lead social worker put it: 


We have support for each individual. The good side is that there’s dedicated 
support. The downside is [the difficulty of] figuring out who to go to when you 
have a challenge: learning community leader, advisor, or social worker? If they 
go to one individual and the sharing isn’t tight, they may go to the person who 
will give them the answer they want. If a Student Services person says, “Let me 
give you something to be helpful,” whereas the [learning community] leader 
says, “You need to be accountable,” there is a bit of tension there. 


The dense social network in learning communities also can be challenging on occasions when 
Student Services staff need to uphold client confidentiality. Without assurance of 
confidentiality, students might be less forthcoming. They reportedly often ask social workers 
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not to share their personal problems with other staff and students. Student Services 
accordingly operates on a “need to know” basis, where the default is to keep issues private. 
This policy sometimes creates tensions when the same students are working with advisors and 
other staff who are eager for information and reassurance that issues are being resolved. 


A third challenge identified by Student Services staff was stress from high workloads. Key 
factors cited here included the small number of staff, the range and changing nature of issues 
young adults face, the stressful nature of work on intense personal problems, the complexity of 
Year Up’s environment, and the multiple roles Student Services staff play in the program. In 
addition to helping to address particular problems and developing connections to community 
resources, these roles include extensive involvement in admissions, onboarding, and ongoing 
learning community activities. 


All offices use master’s in social work (MSW) graduate school interns to augment their Student 
Services staffs. In addition to providing additional capacity at low cost, the internships provide 
valuable experience to MSW students as well as a hiring pipeline for Student Services staff. The 
strategy has its limits, though, as interns do not have the level of skill of a regular MSW staff 
member and so add to supervisory demands on regular staff. 


e Staff cited Year Up’s stipends as indispensable supports for participation. Staff, students, 
and employers believed that tying stipends to compliance with Year Up’s contract 
provided important leverage for fostering positive behaviors. 


Many staff and students believed that Year Up’s stipends played a critical role in making 
participation possible for many students. As one staff member put it: 


| don’t think our students would be able to do it if there wasn’t the stipend; that’s 
really unique, and that’s a huge selling point. Most of our students still have to 
work after school, but it still helps make this time possible. 


Given that stipend amounts are relatively modest, however, staff believed it was important to 
emphasize their purpose was to help with transportation and other additional costs of 
participating in Year Up and not to cover major living expenses. Many students covered living 
expenses by working and/or living at home while attending Year Up. 


Nearly everyone interviewed—staff, participants, and employers—believed the contract was a 
fair and effective strategy for instilling professional attitudes and behaviors. Many liked the way 
it made standards concrete and tied them to real-world consequences. 


Interviewees also liked how Year Up language communicates that students are responsible for 
their own behavior: students “earn” infractions and “fire themselves from the program” if their 
point total hits zero. One staff member explained: 


Given the language [of the contract], the student has incurred and earned the 
infraction. It is not the staff’s fault. It is one of the things | appreciate about the 
organization. | think it is effective in curbing behavior that is career limiting. | 
think the culture work done here is very good. 


Employers liked that the contract fostered accountability, and they appreciated its role in 
screening students prior to internships: 
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What | like is that it’s about self-empowerment and self-accountability. Not that 
anyone is doing something to you. You know the expectations, you’re choosing to 
meet them or not. It’s you deciding not to do it. 


Attitudes towards enforcement varied. Some staff stressed the need to enforce the contract in 
a supportive and non-punitive manner, noting that the ideal was to use the contract to create 

teachable moments with a minimum of actual penalties. Others felt that reluctance to impose 

penalties could undermine the contract’s effectiveness and favored strict enforcement: 


My honest opinion is that there are staff members that need to watch coddling 
the students. We are implementing a program philosophy that this is the 
students’ journey. You cannot win more for them than they can for themselves. If 
Jane Doe needs to fire herself from the program and come back, that may be the 
best thing for her. ... You got to be okay with the student being angry and upset. 
You are preparing them for the corporate world. 


e Analyses of program records indicate that overall contract enforcement was strong, 
though infraction rates varied considerably across offices. 


Although contracts focused on the same behaviors in all offices, approaches to points and 
enforcement varied. Offices varied in the number of initial total points, structured some 
penalties differently (varyingly increasing penalties for repeated infractions, for example), and 
sometimes gave bonus points for weeks with no infractions. One office had a general policy of 
issuing warnings (as “recorded conversations”) in the program’s early weeks to help students 
acclimatize. 


The evaluation team’s analyses of Year Up’s program records found that virtually all students 
(96 percent) received at least one infraction, and 45 percent received 10 or more infractions 
(Exhibit 4-2). The most common reasons were unexcused absences (recorded for 83 percent of 
students), late assignments (80 percent), and excused late arrivals (73 percent). The share 
receiving 10 or more infractions varied from 21 percent to 60 percent across offices. This 
variation likely captures differences in student characteristics and geography (e.g., commuting 
difficulties), as well as in approaches to contract enforcement. 


e Staff cited Year Up’s approach to teaching how to give and receive feedback as one of the 
program’s most potent strategies. 


Year Up’s approach to teaching and practicing the “plus/delta method” in weekly Friday 
Feedback sessions each week was one of its most widely acclaimed strategies in interviews with 
staff, students, and employers. Staff stressed how the program encourages ongoing practice by 
all members of the learning community throughout the week. Program staff often cited the 
high value of these feedback skills in the workplace: 


[Students] learn how to give feedback “up,” a skill they will need to “manage up.” 
Employers don’t want people to just do the work, they want them to think and 
know how to suggest changes and not come off like a jerk. That’s a valuable 
learning and communication skill you really don’t get anywhere else. 
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4.4 Learning at Work: The Internship Phase 


National and local staff emphasized that Year Up guarantees an internship placement to every 
L&D completer. Throughout the year, Year Up staff work with existing and new corporate 
partners to “sell” (in program parlance) the required number of slots. As documented in Section 
4.1, internship sales are a major source of program revenue. 


e Year Up offices succeeded in placing all L&D completers in the PACE treatment group in 
internships, and nearly all (92 percent) interns completed their internships. 


The internship positions closely reflect the occupational fields targeted by Year Up training 
tracks. The most common IT internships involved helpdesk and desktop repair. Financial 
operations positions typically entailed assignments in accounting, portfolio administration, and 
related financial services. Internships in customer services often involved work in call centers 
and junior sales roles. 


e Year Up’s pitch to employers presented internships as a way to give back to society and as 
a source of talented entry-level employees. During PACE, national leaders initiated a 
system-wide shift to put the primary emphasis on meeting customer-specific hiring needs. 


In pitching internships to employers, Year Up historically emphasized both social responsibility 
and corporate self-interest. One staff member described how he varyingly emphasized the two 
messages: “Il present both because you never know what they’Il pick up on, impact to the 
student or impact for the company.” Another staff member agreed: “Our partners don’t 
partner with us for one reason. Some of them see the diversity perspective and see the 
community impact; others just have a really hard time finding talent interested in starting and 
entering at an entry level.” 


Employers often mentioned responding to a mix of the two motives. One initially said, “It’s 
about 90 percent trying to help the community, because we could get contract workers from 
professional staffing agencies. But we’re a part of the city and they’re a part of the city, [so our 
view is] let’s do something good for everybody.” She then went on to enthuse, “They are an 
awesome source of new hires.” Similarly, after describing the benefits of getting involved in the 
community, a manager at a large investment management firm turned to the bottom line: 
“We’re an investment company. If it wasn’t worth it, we wouldn’t do it.” 


During PACE, Year Up’s leaders accelerated efforts to shift internship marketing to a “customer 
solutions” approach. This approach involves working with employers to identify hiring 
difficulties or “pain points” and then customizing recruitment and training to meet these needs. 
Year Up already had such an arrangement at one office, where it provided a substantial flow of 
interns to a large financial services firm. The firm supported 40-50 internship positions per cycle 
and offered jobs to about half of its interns. 


The motivation for a broader shift to this model was the recognition that corporate citizenship 
is difficult to scale. Historically, most Year Up partners were able to host at most a small handful 
of interns. Substantial up-scaling under this model required expanding relationships to a large 
number of companies. From both sales and relationship management standpoints, such 
expansion rapidly becomes difficult and expensive to manage. The customer solutions 
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approach, focusing on fewer key relationships, is more feasible to scale. In principle, 
customization also can improve the fit of interns to hiring needs and improve young adults’ 
opportunities for post-internship employment. 


e Specialized staff in all offices monitored interns closely and provided support as needed. 
Outside mentors provided additional guidance. 


Intern supports begin with close monitoring of each participant’s progress by Year Up local 
office Internship Services managers. Monitoring involves regular—typically weekly— 
communication with interns by email and phone and occasional site visits. There is also regular 
contact with work supervisors, particularly in the early weeks and if supervisors are new to Year 
Up. As described in Chapter 2, Year Up solicits formal input through online surveys of interns 
and worksite supervisors at two points during the internship phase. 


PACE interviews and Year Up statistics suggest that this monitoring and troubleshooting system 
was well implemented. Response rates for Year Up’s online survey were very high—around 90 
percent for both the student and supervisor assessments. Interns and employers consistently 
told PACE interviewers that Year Up staff stayed in close touch and were very responsive as 
needs arose. 


During the internships, interns return to Year Up every Wednesday afternoon to reconnect with 
staff and peers, get help with special challenges, and work on career planning and job search. In 
interviews, national and local staff and students consistently identified a need for more 
structured and valuable activities during these Wednesday sessions. As students become more 
engaged in their internships, their interest in returning to Year Up for these meetings 
reportedly tends to decline. To address the challenges with buy-in, one Year Up office hired a 
new staff member to revamp the approach to Wednesday afternoons and began offering 
supplemental workshop programs on high-interest topics such as preparation for certification 
exams. 


Each local office also runs a mentoring program, assigning each student an outside mentor from 
the community. The purpose is to foster additional connections, support, and career awareness 
beyond Year Up activities. Once paired, Year Up expected students and their mentors to meet 
once a month in person and communicate weekly over email, phone, or text. Year Up offices 
varied in the guidance and direction provided to mentors. Most offices offered initial training, 
but few specified requirements on content and structure once the relationships began. In 
general, Year Up expected that mentors would provide students with career-related advice—on 
internship challenges, professional goals, and finding a job after Year Up graduation—but 
believed that relationships should develop organically. 


e Employers’ experiences with interns were generally positive. 


Employers gave interns high marks in the online survey Year Up conducts towards the end of 
each internship cycle. Nearly all interns either met (55 percent) or exceeded (42 percent) 
managers’ expectations for professional and communication skills, and 66 met and 30 percent 
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exceeded expectations for technical skills.*® Nearly nine in ten (89 percent) said it was likely 
that they would recommend Year Up to a friend or colleague.” 


Similarly, in PACE interviews, employers praised interns’ positive attitudes and professionalism. 
One hospital administrator described an experience setting up IT systems in a new facility: “The 
Year Up kids ran circles around the 45-year-old professionals. They did everything we asked 
them to. They did it with a smile. They did it confidently.” Another employer reported assigning 
Year Up interns to work directly with high-level government clients. A number noted positive 
effects on the climate at work: “They’re eager and excited. They want to learn and engage. It 
helps with our employees as well, keeping up the morale.” 


Several employers compared interns favorably with interns and recent graduates from four- 
year colleges: 


[At Year Up] they do get all the right skills ... especially customer service and 
professionalism. ... Year Up interns hold their own and get hired over the 
stereotypical college graduate from a stable home. Those [college] kids don’t 
know how to interact with a manager. They don’t know how to make small talk 
in an elevator or with a director. They don’t know how to write a professional 
email that doesn’t use text abbreviations, but the Year Up interns do. 


When interns had difficulties at work, employers noted that these often stemmed from outside 
issues affecting attendance or performance. Examples included needing to attend court 
hearings related to ongoing legal problems, medical issues, pregnancy, and homelessness. 


Additional challenges cited by employers include lack of professional dress, etiquette, and 
vocabulary, as well as motivation and attendance. One employer commented that these are the 
same struggles the employer may have with four-year college graduates. For this employer, 
however, the difference is that with Year Up, “once you give feedback, the intern and Year Up 
are trying to make that positive change.” 


4.5 Promoting Transitions to Careers 


Of the 96 percent of treatment group members who enrolled in Year Up, 78 percent graduated 
from the program (Exhibit 4-3). This high completion rate compares favorably with other 
workforce programs with less rigorous requirements. Completion rates were fairly consistent 
across the eight local offices, varying between 75 and 85 percent, except for one office with a 
64-percent rate. 


Efforts to support successful transitions to regular employment begin in the latter half of 
internships, where staff specialists work with students on career planning and pursuing 


38 fs Fl . . - 
Based on calculations from an internal Year Up report (“Historical Survey Analysis”) covering all cohorts served 


during the PACE period (Year Up’s 1/14-7/15 completion cohorts) and based on ratings of about 1,700 interns. 
Statistics include a small number of ratings of Professional Training Corps interns, which were not separated in 


the report. 


° Calculated from Year Up statistics by averaging percent of employers who said they were likely to recommend 


Year Up across core offices (7/14-7/15 completion cohorts). The report defined “likely” as a rating of 7 or 
above on a 10-point scale ranging from not likely at all to extremely likely. 
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employment opportunities with internship and other employers. Intensive job search and 
placement supports extend for four months after graduation. 


e Employment outcomes for alumni were very positive at Year Up’s four-month post- 
graduation follow-up. 


Program data for the PACE sample show that 83 percent of Year Up graduates were working 
full-time (74 percent) or part-time (nine percent) at the program’s four-month post-graduation 
follow-up (Exhibit 4-3). Of those working, 41 percent were at the firms where they had 
interned, and 88 percent were in a Year Up target occupation. Some 77 percent had hourly 
wages at or above Year Up’s $15/hour target wage. The average wage was $17.41 (not shown 
in exhibit). Full-time employment rates were above 70 percent in all but two offices. 


In interviews, local staff and employers noted a strong trend of increasing reliance on short- 
term contracts in hiring in the IT sector. Compared with such arrangements, permanent 
positions typically offer greater job security, advancement possibilities at a firm, and benefits. 
Program data indicate that a majority (57 percent) of Year Up graduates overall held permanent 
positions, ranging from 34 percent to 73 percent across offices. 


Wages were $3.92/hour higher among graduates who worked for their internship employers 
(S$19.70/hour) than among those who did not ($15.78/hour, not shown in exhibit). The 
premium for working in an occupation related to Year Up training, $6.29, was even larger. 
Permanent positions paid slightly less ($16.93) than contract jobs ($18.07). 


e Employers identified a wide variety of factors in decisions on hiring Year Up interns. 


Some firms working with Year Up maintained longstanding college degree requirements for 
career-track positions, whereas others were eager to hire well-trained entry-level workers 
without college degrees. In some instances, experience with Year Up led to the establishment 
of a new layer of entry-level jobs: 


There are some things our senior guys get bored of doing after 10 years, but it is 
exciting for a new person. We crafted a different type of [job] to match the skills 
of [interns] coming in. We got real business value from someone junior coming in 
and getting trained: they contributed to the success of our company. 


For one large financial investment firm, Year Up was an important catalyst for wider retooling 
of entry-level hiring practices: 


Previously, we only brought on people with a traditional four-year degree. ... In 
2005, we weren’t able to find the candidates we were looking for... just from 
four-year schools, so we expanded the market [first, with Year Up interns]. [Year 
Up] is why we’ve started working with the other programs. ... If it wasn’t for the 
success of Year Up, we wouldn’t be expanding to other community organizations. 
These years we bring in about 1,000 new employees a year [mostly from a variety 
of internship programs]. 


Of course, an interest in hiring Year Up interns does not guarantee that particular students and 
employers will be compatible. Notwithstanding intensive screening and training, students may 
not be able to perform at the level required by a given employer. And in-depth interviews 
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documented that some participants did not like the work environment or job possibilities at 
their internship sites (see Section 5.3 in Chapter 5). 


e Although students, Year Up staff, and employers all acknowledged the importance of a 
college degree, only one in six alumni were enrolled in college four months after 
graduating. 


Year Up aims to support college persistence after the program by securing credit for the 
program’s courses at local colleges and through career advising towards the end of the 
program. However, NSC records show that only 16 percent of its graduates were enrolled in 
college in the calendar quarter in which Year Up’s four-month post-program follow-up 
measurement falls (see Exhibit 4-3). About a third (34 percent) of this enrollment was at Year 
Up’s local partner college, and the rest was at other colleges. 


These results affirm that Year Up places a much strong emphasis on employment than on 
continuing education after graduation. Year Up leaders understand the economic forces 
underlying the increasing earnings disadvantage facing workers without college degrees. But 
the program’s core philosophy is that entry-level professional jobs are the best first step in 
career pathways—providing financial means and experience that would better position young 
adults to succeed in college as a concurrent (e.g., part-time) or a later activity. 


Reflecting the tensions between work and school, alumni were less likely to be enrolled in 
college if they were working full-time (12 percent) than if they were working part-time 
(25 percent) or not working (also 25 percent) as of Year Up’s four-month period of post- 
program employment services (not shown in exhibit). 


e During the PACE years, Year Up began to expand its post-program follow-on services. 


As described in Chapter 2, Year Up’s Career and Alumni Service teams actively monitor 
employment outcomes and provide guidance and job search support after graduation. These 
teams also devote substantial effort to developing placement opportunities through existing 
corporate partnerships and wider networks. At the start of the PACE period, post-graduation 
support was limited to four months of such services. 


In 2014, Year Up launched Year Up Professional Resources (YUPRO), an affiliate providing 
career supports to alumni and fee-based talent acquisition services to employers. YUPRO 
maintains an active database of Year Up alumni and position listings. Its financial goals are to be 
self-sustaining, returning any profit to Year Up to support expanded alumni programming. 


Though newly launched around the time PACE cohorts were finishing the program (2014-2015), 
YUPRO was in place for much of the post-program period covered in this initial PACE report and 
may have benefited some members of the study’s treatment group. In the last few years, Year 
Up also has taken steps to extend its own follow-up and employment services from four to 12 
months post-graduation. Some treatment group members also may have benefited from these 
efforts. 
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Exhibit 4-3: Program Completion and Selected Post-Program Outcomes by Office 
Outcome All Offices Office A OfficeB Office C Office D Office E Office F Office G Office H 
Percentage Completing Year Up: 
Of all treatment group members 74.6 61.0 73.8 84.3 73.8 71.9 745 77.3 82.1 
Of those starting L&D 77.7 63.7 78.0 84.8 75.3 79.0 775 79.2 85.3 


Percentage of Graduates Who at the Program’s 


Four-Month Post-Graduation Follow-up Were: 


Working full-time 74.0 63.3 73.8 86.6 73.3 78.6 65.2 74.4 79.7 
Working part-time 9.3 16.3 10.1 5.2 6.7 5.6 10.6 11.2 9.8 
Not working 16.7 20.4 16.0 8.2 20.0 15.9 24.2 14.4 10.5 
Total 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 
Working full-time in target occupation at $15+ hourly 57.9 50.5 53.8 65.7 49.4 62.7 54.1 59.0 75.0 
Enrolled in college during quarter 
At any college 16.2 29.0 18.6 14.9 12.2 14.1 17.4 6.4 18.8 
At a Year Up partner college 5.6 16.0 17 9.7 4.4 6.3 6.3 4.0 1,5 
Percentage of Working Graduates Who Were: 
At internship sponsor firm 41.3 43.6 44.2 34.1 25.7 43.4 40.8 43.9 58.0 
In a Year Up target occupation 88.1 84.6 83.4 91.9 77.1 98.1 91.7 89.7 92.4 
At $15+ hourly (starting wage) 77.3 73.3 70.1 72.3 78.0 78.3 79.6 77.6 92.0 
In permanent (vs. contract) position 68.4 91.8 79.0 72.6 53.0 77.4 68.9 71.0 38.5 
Sample Size 
All graduates 1,245 100 237 134 180 128 207 126 133 
Graduates working 4 months after graduation 1,037 80 199 123 144 108 157 108 119 


SOURCE: PACE calculations based on a December 2016 extract of participant records in Year Up’s Salesforce® management information system. 
NOTES: Offices ordered from smallest to largest impact on the confirmatory earnings outcome (see Exhibit 6-11 in Chapter 6). For sample sizes underlying completion statistics in the 
top panel of this exhibit, see Exhibit 4-2. 


Abt Associates 4. Implementation Analysis |] pg. 57 


Year Up 
Implementation and Early Impact Report 
PACE 


4.6 Summary 


Overall, this analysis finds that Year Up generally implemented its services with high fidelity to 
the program model. Each of the four offices visited for PACE was operating all major Year Up 
components. Quantitative measures attest to strong implementation of major program 
components such as recruitment, retention, internship sales, and post-program employment 
services in all eight local offices. Overall, nearly four in five students completed the program. 


Though all offices implemented each of the program’s major components, there was some 
variation in the strength of implementation. For example, the overall completion rate varied 
from 61 percent (Office A) to 82 percent (Office H). Such differences are not in themselves an 
indication of differential quality of implementation. The offices operated in diverse urban 
environments, and served young adults with varying characteristics. 


Differences in local environments, student characteristics, and operating styles led to different 
approaches to many program services. This variation was to a degree intentional under a 
national management approach that emphasized meeting performance metrics and sought to 
maximize flexible adaptation to local conditions. 


But growth in the number and size of local offices also brought challenges—notably of 
distinguishing functional local innovations from less functional divergences and of minimizing 
costly duplication of effort. Pressures from rapid growth and high performance expectations 
heightened stress and turnover among staff. 


If the program that PACE cohorts experienced in 2013-2014 was well developed, it was by no 
means static. During the PACE years, Year Up expanded its mission, launched the first of 
multiple initiatives to strengthen alignment and standardization, revised its strategy for 
engaging employers, and increased its emphasis on hiring staff with business backgrounds. 
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5. Services Received by Treatment and Control Group Members 


This chapter assesses the degree to which young adults in the study’s treatment and control 
groups received training, support, and employment services and examines their perceptions of 
those services. Quantitative analyses are based on information from 1,301 treatment and 638 
control group members who completed the 18-month follow-up survey. Many comparisons 
focus on the nature of education and training (E&T) experiences and are limited to the subset 
of students in each group who participated in at least some E&T after random assignment. *° 


Qualitative analyses draw on in-depth interviews with 19 treatment and 13 control group 
members randomly chosen from the four local Year Up offices visited for the implementation 
study. Interviews occurred during the latter half of the first year following random assignment, 
when the average treatment group member was in week 16 of the 26-week internship phase. 


Chapter sections correspond to Year Up’s broad phases and services—starting with application 
to the program and moving to instruction, supports, work-based learning experiences, and 
post-program career planning and related services. 


5.1 Applying to Year Up 


PACE sample members’ first experiences with Year Up—during recruitment— preceded random 
assignment and thus were identical for treatment and control group members. This section 
describes sample members’ circumstances prior to applying to Year Up and their motives and 
experiences in applying to the program. It draws on the full in-depth interview sample. 


e Many young adults cited difficulties in prior school and work experiences. 


As documented in Chapter 3 (Exhibit 3-2), Year Up targeted young adults facing a variety of 
disadvantages. Baseline data on the full sample showed that most were black or Hispanic, many 
had struggled in high school and were not currently in college or working full-time, and that 
many recently had experienced financial hardships. Although nearly half (46 percent) had 
attended some college, only three percent had an Associate’s degree or higher. 


In in-depth interviews, young adults cited a variety of reasons for dropping out of college. One 
student dropped out of community college when her parents were deported and she became 
the primary caretaker for her younger siblings. Another ran into difficulties getting a student 
loan after falling into arrears on tuition payments for the prior year. For a number of sample 
members, juggling school and full-time work had been very difficult. 


Many informants described poor experiences at school and work in the past. Difficulties at 
school involved boring classes, lack of motivation, and spells of homelessness and other life 
challenges. One respondent’s description is fairly typical: 


“° Because different subsets of each group participated in E&T, differences in statistics might be affected by 


differences in each group’s characteristics as well as in the kinds of E&T they received. Exhibits 5.1 and 5.2 
present such differences, with statistical tests but without regression adjustment, as they are useful in 
indicating general differences in experiences. The exhibits also include several measures that are measured for 
the full treatment and control group samples. In these instances, shown as non-italicized rows, estimated 
differences do represent experimental impacts and are regression-adjusted. 
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I’m someone who’s really curious. | like learning, but don’t feel school did it 
correct. ... | didn’t feel any motivation. It was dry all the time. Either too fast or 
too slow, not individualized, just this place I’m forced to go to every day. Didn’t 
seem | had any control over where I’m going. 


Uninteresting work experiences in low-wage jobs also fostered a desire for change: “There 
[wasn’t] much mental stimulation there ... besides it was incredibly physically laborious. ... | did 
not like it. It told me that low jobs were terrible. It told me that.” 


e Students cited different aspects of Year Up as resonating as they learned about the 
program. 


Given past struggles in high school and college, Year Up’s appeal for many was how it seemed 
to address the problematic aspects of past school and work environments. As one participant 
put it, “It was an alternative, and | knew that traditional school was not quite my cup of tea. ... | 
felt it might actually let me enjoy learning.” 


Many students cited Year Up’s offer of college credits and financial stipends as a strong draw: 
“It interested me because they said that you can go to school for free and get paid. | thought it 
was unreal. | was like, ‘This is a win-win-win.’” 


The sense of community at Year Up appealed to some applicants. As one said, “The main thing 
was that | could be part of something, not so alone. [It] gave me the sense | could be part of 
something again with the internship and get a job afterwards.” 


Some informants said that strong interests in technology made them excited about the chance 
to train for jobs in the IT sector. Others liked the opportunity to get connected with big, 
prestigious employers. As one put it: “[I liked] the fact that they had connections to jobs [at 
those places]—like the White House, AOL, BET—just some of those names. Even if you don’t 
stay working there, just having some of those names [on your resume]—they’Il catch peoples’ 


n” 


eye. 


For some applicants, the most important motivator was simply the offer of a route out of their 
current situation. One informant described her reaction to Year Up thusly: “Why not? I’m 
working at Walmart, I’m not really doing anything... [T]his is a good opportunity to... create a 
better future.” 


5.2. Receipt of Education and Training Services 


This section examines participation in employment and services by young adults in the 
evaluation’s randomly assigned treatment and control groups. Exhibit 5-1 provides a quick 
primer on reading impact tables in this report. As the exhibit explains, some comparisons 
involve the full sample and represent experimental impacts, whereas others (italicized rows in 
Exhibits 5-2 and 5-3) apply to sample members who received education and training and are 
non-experimental. 
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Exhibit 5-1: How to Read Impact Tables 


This box provides a brief guide to reading impact tables in this chapter (Exhibits 5-2 and 5-3) and Chapter 6. In these tables, 
the first (left-most) column identifies the Outcome for which results appear in subsequent columns and notes the 
measurement units for outcomes and impacts. The next column presents the average outcome value for the Treatment 
group, and the following column gives the average value for the Control group. The next column -- Impact (Difference) -- 
shows the impact, or difference between the treatment and control group means. 


Statistics in non-italicized rows apply to the full survey sample: differences estimate “experimental” impacts and are 
“regression-adjusted.” As explained in Chapter 3 and Technical Appendix A, regression adjustment helps to correct for any 
small differences arising by chance during random assignment. Statistics in italicized rows apply to a subset of treatment and 
control group members. Because the latter groups are no longer comparable, the statistics represent simple descriptive 
comparisons (rather than experimental impacts) and are not regression-adjusted. 


The next column, Standard Error, is a measure of uncertainty in the estimated impact (difference) that reflects both chance 
variation due to randomization and any measurement error. 


The final column, p-Value, is the probability that the observed or larger difference between the treatment and control group 
values would occur by chance, even if there was no difference in the outcomes of the two groups. There are several common 
standards for judging statistical significance. In this report, tests are considered statistically significant and highlighted in 
tables (with one or more asterisks) if the p-value is less than or equal to .10. Tests with smaller p-values are separately 
flagged: 


*for .10 


** for .05 
ST ORON 


Analyses of administrative data in Chapter 4 showed that nearly all treatment group members 
(96 percent) started the L&D phase of the program. In the 18-month follow-up survey, close to 
100 percent of treatment group respondents reported participating in Year Up. A more generic 
survey question on education and training experiences after random assignment produced a 
lower estimate (79 percent, see Exhibit 5.2), possibly due to confusion about dates or 
respondents not thinking of Year Up as a typical instructional program. 


More than half (57 percent) of control group members also reported attending school between 
random assignment and the 18-month survey. Most of the control group members who took 
courses did so at a two-year (58 percent) or four-year (20 percent) college (not shown in 
exhibit). 


e Treatment group members were more likely to cite promising instructional practices than 
were control group members who attended other programs. 


Among respondents in each group with some postsecondary education during the follow-up 
period, treatment group members were 8 to 16 percentage points more likely than control 
group members to say that “most” or “all” of class time was spent on project- and group-based 
work, and eight points /ess likely to say that “most” or “all” of class time was spent on lectures 
(Exhibit 5-2). Nearly twice as many treatment (56 percent) as control (29 percent) group 
members receiving education and training said that their classes incorporated active learning 
methods most or all of the time, and more treatment (69 percent) than control (51 percent) 
group members agreed that courses they took were relevant to their life or career. 


Treatment group members who completed in-depth interviews similarly described Year Up 
classes as more hands-on, interactive, and applicable to the real world than their previous 
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experiences in school. One young adult said, “There’s more interaction than lecture. You can 
talk more, be more communicative. ... It’s easier interacting with other students. | [also liked] 
that it was more hands on.” Another described how their IT class approached group work: “For 
my group, they gave us two really technical people and then threw in a couple of us that 
weren't as technical—so it was easy to learn from them. ... They pushed for us to be self-reliant, 
which | really liked.” 


Exhibit 5-2: Education and Training Experiences by Treatment-Control Status 


p-Value 
Impact Standard (Two- 
Outcome Treatment Control (Difference) Error Sided) 
Percentage receiving education and training after random 793 56.6 499.6 09 <.001 
assignment 
Among respondents receiving E&T, the percentage 
Reporting that classes at first place attended spent 
mosvall time on: 
Lectures 33.9 41.8 -8.0""* 3.0 .008 
Group discussion 43.0 30.6 +12.4*** 29 <.001 
Group projects 30.2 14.6 +15.6*** 23 <.001 
Individual projects 25.4 17.2 +8,2*** 24 <.001 
Indicating that classes at first place often used active 564 285 428.0 28 <001 
learning methods 
Strongly agreeing that classes at first place were 
pad eye ° ane one all on ce 
Ever taking any course in life skills 76.3 32.0 +44, 3*** 27 <.001 
Ever taking a life skills course paying a great deal of 
attention to: 
Study skills 45.7 16.4 +29,4*** 25 <.001 
Critical thinking 54.5 18.4 +36.1"* 25 <.001 
Time management 57.3 16.6 +40,8*** 25 <.001 
Managing stress/anger 36.6 9.1 +27,5*** 21 <.001 
Staying motivated 56.8 17.0 +39, 7*** 25 <.001 
Acting professionally 70.1 21.0 +49, 1*** 25 <.001 
Communicating well 65.8 18.2 +47.6*** 25 <.001 
Working in groups 54.2 13.5 +40,7*** 24 <.001 
Managing finances 43.3 8.3 +35.0"** 21 <.001 
Career planning 53.5 15.4 +38,.1"** 2.5 <.001 
Job search 44.3 8.6 +35, 7°*** 21 <.001 
Dealing with other life problems 43.5 11.6 +31.8"* 2.3 <.001 


SOURCE: PACE 18-month follow-up survey. 

NOTES: Differences statistically significant: * at the 10% level, ** at the 5% level, *** at the 1% level (two-tailed tests). 

Where not italicized, outcomes apply to the full survey sample (1,301 treatment and 638 control group members): differences 
estimate experimental impacts and are regression-adjusted. Italicized outcomes apply to the subset of survey respondents attending 
education and training after random assignment (1,035 treatment and 367 control group members). Treatment-control differences 
for italicized outcomes are simple descriptive comparisons and not regression-adjusted. 
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e Exposure to life skills courses was much more common at Year Up than at institutions 
attended by control group members. 


As described in Chapter 2, Year Up’s Pro Skills course aims to instill skills for success in 
professional settings. The 18-month follow-up survey asked respondents who had received E&T 
if they had taken courses in skills in “how to succeed at school, work, or other areas of life.” A 
substantially higher fraction of treatment (76 percent) than control (32 percent) group 
members answered affirmatively (see Exhibit 5.2).77 


When asked about instruction in a series specific life skills, substantially more treatment than 
control group members said they took courses paying a great deal of attention to each skill. The 
largest treatment-control differences were for instruction in acting professionally (49 
percentage points), communicating well (48 percentage points), time management (41 
percentage points), and working in groups (41 percentage points). 


In in-depth interviews, treatment group members also said they had learned useful professional 
skills at Year Up. They credited a mix of influences at Year Up, emphasizing experiences with the 
contract and “Year Up norms.” One student described the progress she’d made in these terms: 


I’m on time to classes. Dress more professional. Even on weekends | have to go to 
work, but | make sure everything is ironed and I’m looking up to par. I’m 
respecting peoples’ opinion more. | know how to network with people now and 
talk to different people about stuff. I’m not as shy as | used to be. ... Now, I'll talk 
to who-ever. 


Many interviewees believed that Year Up had provided effective preparation for operating ina 
corporate environment. One commented: “I had no idea... what the corporate world is like: 
how to behave in social events, what’s considered acceptable, what’s not.” 


Informants emphasized the value of specific skills and strategies for networking. For example, 
one described how he had prepared for an interview through online research on the CEO and 
director of IT and organizing questions for the interview. He had learned about the importance 
of questions early on at Year Up: “Way back during my [Year Up admission] interview, they 
asked me did | have any questions for them. | said no. | thought that was the correct answer. 
Little did | know it’s not the correct answer.” 


5.3. Supports for Education and Training 


Chapters 2 and 4 described the extensive services Year Up provides to support learning and 
program retention. Treatment group members consistently reported receiving higher levels of 
support at Year Up than control group members reported for other E&T programs (Exhibit 5.3). 


Treatment group members were more likely than control group members to receive career 
counseling (by 33 percentage points), help arranging supports when needed (25 percentage 
points), and tutoring (18 percentage points) at the first place of instruction they attended after 
random assignment (i.e., Year Up for treatment group members). Year Up students also were 


“* Because all Year Up students take Pro Skills, the 76-percent figure implies that survey reports are subject to 


significant underreporting of life skills instruction. 


Abt Associates 5. Services Received by Treatment and Control Group Members 1 pg. 63 


Year Up 
Implementation and Early Impact Report 
PACE 


more likely to receive academic advising, financial aid advising, and grants or scholarships. 
Treatment group members enrolling in Year Up were less likely to take out loans than were 
control group members who enrolled in other education programs. In the entire sample, 
treatment group members were nine percentage points less likely than control group members 
to identify financial need as a challenge to enrolling or persisting in school. 


In addition to providing services, creating a supportive community is an important goal in Year 
Up’s “high support, high expectations” model. When asked about the emphasis on “being part 
of a community with other students, instructors, and staff” at the first place attended after 
random assignment, 71 percent of treatment group members, but only 40 percent of control 
group members, reported a great deal of emphasis on community. 


Exhibit 5-3: Supports Received by Treatment-Control Status 


Impact Standard 
Outcome Treatment Control (Difference) Error p-Value 


Among respondents receiving E&T, percentage who at 
Year Up/first place of instruction ever received: 


Career counseling 58.4 25.7 +32,8*** 2.8 <.001 
Academic advising 55.7 47.1 +8.6*** 3.1 0.005 
Financial aid advising 41.4 32.5 +8,.9*** 2.9 0.002 
Tutoring 43.9 26.3 +177" 2.8 <.001 
Help arranging supports for school or work 36.3 10.8 +25, 5*** 2.2 <.001 
Grants/scholarship 80.7 67.3 +13,5*** 2.8 <.001 
Loan 6.1 27.9 21.7" 2.5 <.001 


Among respondents receiving E&T, percentage who at 
Year Up/first place of instruction: 


Perceived a great deal of emphasis on community 70.5 39.9 +30. 7*** 3.0 <.001 
Participated in a work experience activity 81.4 44.1 +37, 3*** 2.9 <.001 
Received job search or placement assistance 60.7 27.5 +38.2*** 2.7 <.001 


Among all respondents, percentage: 
Citing financial support as a challenge in school 


51.9 60.9 -9.0""* 2.3 <,001 
enrollment or persistence 
Ever receiving help with: 
Arranging supports to meet school, work, or family 43.7 113 430 Ger 18 <001 
responsibilities 
Career counseling 58.7 19.7 +39.0*** 2.0 <.001 
Job search or placement 61.5 18.3 +43.2*** 2.0 <.001 


SOURCE: PACE 18-month follow-up survey. 

NOTES: Differences statistically significant: * at the 10% level, ** at the 5% level, *** at the 1% level (two-tailed tests). Where not 
italicized, outcomes apply to the full survey sample (1,301 treatment and 638 control group members): differences estimate 
experimental impacts and are regression-adjusted. Italicized outcomes apply to the subset of survey respondents attending 
education and training after random assignment (1,035 treatment and 367 control group members). Treatment-control differences 
for italicized outcomes are simple descriptive comparisons and not regression-adjusted. 


More generally, the 18-month survey asked all respondents whether they had received 
supports from any organization to help meet school, work, or family responsibilities. The 
bottom panel of Exhibit 5.3 shows a substantial positive impact (32 percentage points) on 
receipt of such supports. 
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e Students cited Year Up’s initial orientation session and instruction in “feedback” skills as 
especially valuable. 


One student described orientation as “one of the most important weeks” in the program 
because “you get to know everybody around you. | feel like that was what really established the 
whole [feeling of], ‘Okay, we’re in this together, we’re gonna do this. We’ve got each other’s 
back.’” As an example of how orientation activities had helped in getting to know instructors, 
this respondent described a “teacher museum” in which each instructor brought in three 
objects that represented him or her, and students had to guess the owner. “You really got to 
know the teachers on a more personal level before the program even started, so | felt like that 
was a good foundation for us.” 


Many students said that training in feedback during Friday Feedback sessions was one of the 
most valuable skills they had learned in Year Up. 


One thing that | really, really like about Year Up is the way they do feedback. ... | 
have never had feedback that way. They focus on your strengths and commend 
you for what you did right and your strong points. Then they say, “With that in 
mind, | would encourage you to do this.” See? So it’s not saying, “But you do this 
all wrong.” They say it in a way that’s going to encourage somebody to keep that 
growth area in mind. | like that a lot. 


Students described how feedback received during Year Up helped them to strengthen 
important skills. One said: “I had a lot of feedback on the way | talked. They wanted me to 
project my voice. Be more active in speaking in a group setting. Or speaking in front of people. | 
felt that | have improved in that.” Another noted how they received useful feedback through 
multiple channels: “If anything needs to be addressed, someone will pull you, one by one, and 
tell you. There’s... advisors... They give feedback... My mentor’s given me feedback about my 
life choices. All throughout there’s feedback.” 


e Most participants interviewed felt Year Up’s contract was useful and fair, though one felt 
it imposed too much stress. 


Most in-depth interviewees understood and approved of the purpose of the contract: 


| like it. | think it holds every student accountable. If they didn’t have it, 

everybody’d be doing what they wanted. | don’t think that would be fair. Now 

that we’re on internship, we’re dressed professional because of Year Up... not 

using slang, getting to work on time, actually turning our work in on time too. | 

like it. 
Another respondent cited positive outcomes from having been put on a contract amendment 
plan (a form of warning or probation) when he “didn’t have the right mentality.” He said: “That 
helped light the fire under me. Made me realize that, oh, man, I’m slipping. | don’t wanna lose 
this opportunity.” 


One informant’s comments captured the experience of the students who had the most 
difficulty meeting the program’s expectations. He believed that the contract had made learning 
difficult because of a constant fear of being docked points. “Every time you see the points drop, 
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you’re just more worried about that.” Looking ahead, he did not have much confidence: “I don’t 
feel determined enough to actually go into internship and feel I’ll ... be able to [handle 
assignments] with confidence.” 


5.4 Learning from Work 


Year Up’s six-month internships afford opportunities to apply and extend skills learned during 
the preceding L&D phase and get to know potential employer and sources of job references. 
Among survey respondents who received E&T, nearly two times as many treatment (81 
percent) as control (44 percent) group members said they were offered opportunities for direct 
experience in occupations related to their studies or career goals (see Exhibit 5-3).”7 


e Most students reported learning a great deal during internships. 


When asked about skills learned from internships, some in-depth interviewees emphasized 
professional skills and others emphasized technical knowledge. One intern said he learned a lot 
about time management: “Your supervisor, depending on what his personality, and the way he 
runs it, most of the time he’s not gonna chase you down... You had a five-day limit. You go over 
those five days, then they’ll say, ‘Did you get this done?” Another intern emphasized the 
technical skills she acquired: “I can really build a computer from the ground up and have it 
working... that’s definitely something that came in handy.” 


One intern described an environment that rewarded initiative: 


It’s exciting—I like it. I’ve learned a lot. ... | hardly knew about SharePoint. Right 
now, | could say I’m the go-to gal. My manager, she’s pretty impressed that | 
caught on so quickly. I’ve been practicing every day. If [my manager] wants to 
change something, | have to figure it out. Ill just Google it and do it. 


Many informants reported that they were thriving in internships. One said, “I love my team. | 
think it’s a great—it’s very close. There’s only 10 of us. | like the environment.” Another 
enthused: “I would absolutely love to stay there. | could definitely see myself there in five 
years—obviously in a higher position—and continuing to evolve.” One young woman said 
although she initially was very anxious (“What if | don’t fit in? What if | don’t know what I’m 
doing? What if | bring the whole company down?”), the experience had been positive: “I was 
definitely placed on the right team. ... We all get along. We get work done and we have fun, 
too.” 


e Less-positive internship experiences often entailed less-substantive work, too much 
downtime, or too little interaction with co-workers. 


Some interviewees found their internship environments less engaging than anticipated. One 
intern had expected his internship to be more exciting and “energy filled” because it was “really 
hyped up” during L&D, but found himself “[just working] in a cubicle for eight hours... “ 


“The 81 percent figure is very close to the 85 percent of enrollees completing the L&D phase to advance to 


internships according to Year Up administrative records. 
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One intern described her IT position as too quiet. “[It is] a big change from Year Up: People are 
busy, don’t interact ... you’re kinda to yourself a lot of the time. I’m an intern, so | don’t get jobs 
like everybody else. | get a lot of downtime.” The quiet environment made his “mind wander.” 


Downtime was a theme in several informants’ feedback on internships. In some offices, Year Up 
staff address this issue by preparing students to provide feedback to their supervisors and take 
initiative in identifying additional tasks. For one student, this feedback led to an arrangement in 
which he learned Java programming with guidance from his supervisor. 


5.5 Transitioning to Careers 


The emphasis on career planning and job search/placement at Year Up intensifies towards the 
end of the internship and extends for up to four months after graduation. 


e Treatment group members were far more likely to receive career planning and 
employment services than control group members. 


Compared with control group members, treatment group members were 39 percentage points 
more likely in the 18-month survey to report ever receiving career counseling after random 
assignment (see last panel in Exhibit 5.3). Similarly, the percentage receiving job search or 
placement assistance was 43 percentage points higher for treatment than for control group 
members. 


In in-depth interviews conducted during internships, treatment group members also cited 
internship experience as influential in their career plans. As one intern put it, “Il want to get a 
software engineering special degree, a master’s degree, because | want to continue this 
programming. | really like programming. I’m passionate.” Similarly, another explained how his 
internship had helped to crystalize his career plans: 


| like working in IT because it’s challenging; it’s always a new problem...! like 
fixing people’s problems. | like when they’re happy after I’ve fixed what their 
issue was and stuff. 


Less-positive experiences also helped some students to identify work environments to avoid or 
identify career interests beyond Year Up’s focal occupations (mostly in IT and finance). In a 
number of instances, however, these students saw their new skills as enabling them to pay the 
bills in the short-run while working on a college degree in another field. 


e Control group members also had strong career aspirations, but generally had to find the 
path forward on their own. 


Given that Year Up screens applicants for motivation, it is not surprising that control group 
members also had high hopes. Some found their way to other workforce programs, while 
others uncovered other sources of support. 


One young man’s story indicates how good luck can arise in the absence of workforce 
programs. Things were difficult when he applied to Year Up. After struggling to find direction in 
college, he had dropped out with $11,000 in debt and stayed home, struggling with depression 
and caring for a sick father for two years. Then he landed a job as production scheduler, earning 
$19.60 an hour with good benefits, at a local factory where his mother worked. The firm proved 
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to have a very supportive outlook in supporting training and job advancement. The plant 
manager took an interest and helped him map out a career plan. The plan included steps to 
remove a transcript hold for debt arrears at one college and enroll in an Operational 
Management program at another. He summed up the catalytic role his new job had played as 
follows: “It’s like everything has fallen into place. | finally got a car—that’s a huge blessing. [The 
job has] given me a sense of purpose, made me focus. It’s like now | honestly can start life.” 


5.6 Summary 


Analyses in this chapter show that treatment group members received higher levels of training 
and a variety of supports than control group members. In interviews, Year Up participants 
described many positive experiences with the program’s instruction, supports, and internships. 


In contrast, about half of control group members received some E&T, mostly in local college 
programs. They generally had more difficulties charting career directions in these programs, 
although motivation and luck helped some find their way to career pathways. 
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6. Year Up’s Early Impacts 


Evidence that Year Up implemented its program with high fidelity (Chapter 4) and substantially 
increased service receipt (Chapter 5) puts the impact analysis in a strong position to test the 
program’s theory of change. This chapter presents impact estimates for a range of outcomes, 
including: earnings, employment, and related measures of career progress (Section 6.1); 
postsecondary education (Section 6.2); and other outcomes, including financial status, psycho- 
social skills, and family formation (Section 6.3). The last section (6.4) provides analyses of 
differences in impacts for key outcomes across subgroups and local offices. 


6.1 Impacts on Earnings, Employment, and Career Progress 


This section presents the report’s main impact findings on employment and earnings. It looks 
first at results for the pre-specified confirmatory test of Year Up’s effectiveness: whether the 
program affected students’ average quarterly earnings in the 6" and 7” quarters following the 
quarter of random assignment (Quarter 0). Subsequent analyses examine the time path of 
earnings and employment impacts, the types of jobs participants gained, and the degree to 
which they believed they were making progress in a career. Measures of average earnings in 
the impact analysis include sample members who did not work (and thus had zero earnings) 
during the relevant quarters. 


e Year Up produced a large positive impact on the confirmatory outcome—average 
earnings in Quarters 6 and 7 after random assignment. 


Based on data from the National Directory of New Hires (NDNH), average quarterly earnings 
were $1,895 higher (53 percent) for treatment group members ($5,454) than for control group 
members ($3,559), as seen in the first panel of Exhibit 6-1. This impact is the largest reported to 
date in random assignment studies of programs aiming to boost earnings among low-income 
youth and adults (see Chapter 1). 


Data from the 18-month follow-up survey provide a very similar impact estimate (as seen in the 
third panel): $1,970, a 50 percent increase.*’ The small difference between impact estimates 
from NDNH and the survey ($75) may result in part from underreporting of wage records to the 
federal NDNH database by Washington State.“ To confirm this suspicion, we used survey data 
for Year 


“The survey figures convert the available survey measure—earnings in the prior week (calculated as hourly 


wage times number of hours worked)—to a calendar quarter-level estimate by multiplying by 13 (the average 
number of weeks in a quarter). 

The Abt team identified two calendar quarters (2014 Q4 and 2015 Q2) in Washington State with lower NDNH 
record counts than expected based on counts in adjacent quarters. The latter of these quarters affected 
Quarter 7 earnings for one of two cohorts enrolled in Year Up’s Washington office. 
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Exhibit 6-1: Impacts on Earnings and Employment 
Treatment Control Impact Standard _ Relative 
Outcome Group Group (Difference) Error Impact _p-Value 
Confirmatory Outcome | 
oo SAS QUAN aed 3,559 +1895"* (143) 53.2% =~ <.001 
Sample size 1,638 858 
Secondary Outcomes | 
Average total earnings in successive 
follow-up years ($) 
Year 1 (quarters 0-3) 3,853 9,191 -5,338 ** (238) -58.1 <.001 
Year 2 (quarters 4-7) 18,435 13,254 +5,181"* (474) 39.1 <.001 
Year 3 (quarters 8-11) 24,422 17,400 +7,011"* (619) 40.3 <.001 
Sample size 1,638 858 
Exploratory Outcomes | 
Average quarterly earnings per follow-up 5,907 3,937 +1,970** (183) 50.0 <.001 
survey ($) 
Hours worked last week (%) 
Not currently employed 26.0 26.5 -0.4 (2.1) 1.5 0.829 
1-19 hours 2.7 8.4 =5.1 (1.2) -67.9 <.001 
20-34 hours 12.4 23.0 -10.6** (1.9) -46.1 <.001 
35+ hours 58.8 42.0 +16.8** (2.3) 40.0 <.001 
Total 100.0 10.0 Ts 
Average weekly hours 27.9 24.4 +3.4** (.8) 13.9 <.001 
Hourly wages if employed (%) 
1-9 dollars 7.9 19.0 -11.1" (1.7) -58.4 <.001 
10-14 dollars 18.9 39.1 -20.2*** (2.2) -51.7 <.001 
15-19 dollars 26.2 9.9 +16.3* (1.7) 164.6 <.001 
20+ dollars 19.7 47 +15.0** (1.4) 319.1 <.001 
Total 100.0 100.0 
Average hourly wage ($) 16.06 12.20 3.87 *** (0.24) 31.7 <.001 
Sample size (all survey respondents) 1,301 638 


SOURCES: Confirmatory and secondary earnings outcomes are based on quarterly wage records obtained in a match to the 

National Directory of New Hires. Exploratory outcomes are based on the PACE 18-month follow-up survey. 

NOTES: Statistically significant: * at the 10% level, ** at the 5% level, *** at the 1% level. 

Significance tests are one-sided for confirmatory outcome and total earnings in Years 2 and 3. Tests are two-sided for Year 1 

earnings and other exploratory outcomes. Statistics under “Relative Impact” represent impacts in column 3 as a fraction of the 
corresponding control group mean (i.e., 100 * [impact/control group mean)). 


Up’s Seattle office to adjust the corresponding NDNH estimate.” So adjusted, the overall NDNH 
($1,990) and survey ($1,970) impacts were nearly identical, suggesting that Washington data 
explain most of the NDNH-survey discrepancy.*° 


“The adjustment applied a two-step procedure to office-level statistics. The first step was to run an ordinary 


least squares regression on point estimates for impacts from the seven non-Washington offices. Specifically, 
analysts regressed NDNH-based impacts on the corresponding survey-based estimates for each office (R’= 
89%). The second step involved multiplying the survey-based impact estimate for Washington by the 
regression coefficient relating survey to NDNH estimates in the remaining offices. The result was an adjusted 
NDNH estimate for Washington. The same procedure was used to estimate mean NDNH earnings for the 
Washington control group. 
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e The time path of earnings impacts shows the expected pattern: reduced earnings during 
the program year followed by higher earnings once the program ended. Extended analysis 
shows that earnings impacts remained substantial through the end of the third follow-up 
year. 


Treatment group members substantially reduced their work effort early in the follow-up period 
to meet the demands of the full-time Year Up program—a common finding in workforce 
evaluations. As seen in Exhibit 6-2, they earned substantially less than their control group 
counterparts in Quarters 1-3 (and somewhat less in Quarters 0 and 4 when they began and 
completed the program, respectively). Fewer treatment than control group members worked 
during these quarters (Exhibit 6-3), and those who did work earned substantially less—likely 
because they worked fewer hours.” 


Large positive earnings impacts appear in Quarter 5 and persist through Quarter 11 (the end of 
the third follow-up year). 


For employment, small positive impacts emerge in the second year but disappear in the third 
(Exhibit 6-3). The finding of much larger proportionate earnings impacts than employment 
impacts implies that treatment group members were earning more because of higher wages or 
more hours rather than because more had jobs. 


Survey-based estimates for hours and wages in the prior week support this inference (see last 
panel of Exhibit 6-1). Year Up produced a 3.4-hour increase in hours worked (a 14-percent 
effect), and the treatment group was more likely than the control group to work full-time 

(59 percent vs. 42 percent). The program increased hourly wages by $3.87—a 32-percent 
effect. There was no impact on the fraction with any employment in the prior week: in both the 
treatment and control groups, about 74 percent reported working at least some hours. *® 


“6 As another check on the NDNH-based estimate, we re-ran impacts for the confirmatory earnings outcome 


excluding sample members from Year Up’s Seattle site (10 percent of the full sample). The resulting impact 


estimate ($1,932) also is very similar to the adjusted full-sample ($1,990) and survey ($1,970) estimates. 


“7 For example, in Quarter 3, the average treatment group member who worked earned $2,361 (dividing total 


earnings by the fraction worked—$765/.324), compared with $3,789 for control group members 
($2,766/.730). 

NDNH-based employment estimates are slightly higher than survey-based estimates for treatment group 
members but nearly identical for control group members. The difference could be a chance result from 
sampling error or a result of differences in time periods covered by each measure. The NDNH-based measure 
captures quarterly earnings, while the survey-based estimate uses a single week’s worth of earnings to 
estimate quarterly earnings (e.g., multiplying by 13, the average number of weeks in a quarter). NDNH thus 
reflects the cumulative effect of brief jobs to a greater extent than the (weekly-based) survey figure. If 
treatment group members were more likely to have very short job spells after graduation than control group 
members had in the same follow-up period, we would expect to see higher fractions of the former employed 
in NDNH than in the survey but smaller differences in average earnings (because the additional jobs were very 
brief). Higher job turnover is quite plausible given the strong drive to help participants find jobs following Year 
Up graduation and anecdotal reports that multiple offers and switches were not uncommon. Supporting this 
interpretation, employment impacts fade while substantial earnings impacts persist in Quarters 8-9. 
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Exhibit 6-2: Impact on Average Earnings in Successive Follow-Up Quarters 
Average Quarterly Earnings ($) 
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SOURCE: Match to wage records in the National Directory of New Hires for 1,638 treatment and 858 control group members. 
NOTES: Statistically significant in a two-tailed test: * at the 10-percent level, ** at the 5-percent level, *** at the 1-percent level. 


Exhibit 6-3: Impact on Employment in Successive Follow-Up Quarters 
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SOURCE: Match to wage records in the National Directory of New Hires for 1,638 treatment and 858 control group members. 
NOTES: Statistically significant in a two-tailed test: * at the 10-percent level, ** at the 5-percent level, *** at the 1-percent level. 


e Negative earnings impacts associated with Year Up participation in Year 1 are more than 
offset by positive earnings impacts in Years 2 and 3 and program stipends during Year 1. 


Summing quarters for the first follow-up year (the program year), earnings impacts indicate 
that treatment group members reduced employment to participate in Year Up. Average 
earnings were substantially lower (-$5,338) for treatment than for control group members 
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(Exhibit 6-1, middle panel). Impacts turn positive thereafter: the treatment-control difference is 
large and positive in follow-up Years 2 ($5,181, a 39 percent impact) and 3 ($7,011, a 40 
percent impact). The total positive impact for Years 2 and 3—$12,192—is more than twice the 
size of treatment group members’ net loss in Year 1. 


Moreover, Year Up stipends averaging $7,172 per treatment group member (and not reported 
as NDNH earnings) more than made up for the -$5,338 impact on regular earnings in follow-up 
Year 1.79 


By comparison, the largest impacts in the existing literature are smaller in both absolute and 
proportionate terms. As summarized in Chapter 1, the SEIS evaluation reported earnings 
impacts of $4,011 (a 29-percent increase) for the three sectoral programs pooled and $4,663 (a 
32-percent increase) for the most successful site—Per Scholas (Maguire et al. 2010). The pooled 
SEIS estimate for 18- to 24-year-olds was smaller, at $3,092 (25 percent). Project QUEST’s 
impacts for all adults reached $5,080 (22 percent) in the sixth follow-up year, but this program 
had no positive impact for young adults (Elliot & Roder 2017). 


e Substantially more treatment than control group members were working in career-track 
jobs as of the 18-month follow-up. 


Results strongly support the evaluation’s secondary hypotheses for increased career-track 
employment (see Exhibit 6-4). Year Up generated substantial increases in the percentages of 
graduates working in jobs that paid at least $15/hour (31 percentage points), that required at 
least mid-level skills (28 percentage points), and that were in Year Up target occupations 

(28 percentage points). It also increased graduates’ perceptions of being on a good track 
career-wise, with modest positive effects on self-assessed career progress, confidence in career 
knowledge, and career networks (effect sizes of .28, .18, and .34, respectively). 


Exhibit 6-4: Impacts on Selected Career Outcomes 


Treatment Control Impact Standard Relative p- 
Outcome Group Group _ (Difference) Error Impact Value 
Indicators of career pathways employment (%) 
Working and $15/hour or more 45.8 14.6 +312" (1.9) 213.7 <.001 
Working in job requiring at least mid-level skills 43.4 15.5 21.9" (2.0) 180.0 <.001 
Working in a Year Up target occupation 46.2 18.0 +28.2*** (2.0) 156.7 <.001 
Indices of self-assessed career development 
Perceived career progress 3.51 3.33 +0,18*** (0.03) +0.28 <.001 
Confidence in career knowledge 3.26 3.15 +0,10*** (0.03) +0.18 <.001 
Access to career network 1.78 1.68 +0.10*** (0.01) +0.34 <.001 


Sample siz 131 63 (is 


SOURCE: PACE 18-month follow-up survey. 

NOTES: Statistically significant in a one-tailed test: * at the 10% level; ** at the 5% level, *** at the 1% level. Statistics in the Relative 
Impacts column are relative impacts for indicators of career pathways employment (100*[impact/control group mean]) and effect 
sizes for career development indices (impact/standard deviation for control group). See Appendix A for definitions of outcome 
measures. 


“The $7,172 for PACE cohorts differs slightly from the $6,614 figure in Exhibit 4-1 of Chapter 4, which is based 


on Year Up financial data for all four cohorts each office enrolled between 2013 and 2014. The PACE sample 
involved only two cohorts, which enrolled at different points in different offices during the same period (see 
Chapter 3, Section 3.3.1). 
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Distributions by occupation were quite different for treatment and control group members 
employed at the time of the survey (Exhibit 6-5).°° Substantially more treatment than control 
group members were working in information technology and computer-related fields (37 
percent vs. four percent) and business and financial services (23 percent vs. 14 percent), the 
two main occupational sectors Year Up targets. Fewer treatment than control group members 
were in sales (eight percent vs. 19 percent), food preparation and service (five percent vs. 

13 percent), and transportation and material moving (three percent vs. nine percent) 
occupations. 


Exhibit 6-5: Percentage Employed in Broad Occupational Sectors 
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SOURCE: PACE 18-month follow-up survey. 
NOTES: Based on samples of 954 treatment and 460 control group members reporting employment at the time of the survey. 
Statistics incorporate non-response weights but are not regression-adjusted. 


6.2. Impacts on Postsecondary Education 


Analyses in this section trace impacts on college enrollment for up to 11 follow-up quarters 
after random assignment based on college records from the NSC. The section also presents 
impact estimates for receipt of college credits and of credentials of different types, as 
measured in the 18-month follow-up survey. Given uncertainty about post-program effects, the 
analysis treats education hypotheses as exploratory and applies two-tailed tests. 


°° Exhibit 6-5 provides simple descriptive comparisons for respondents in each occupational group who were 


working at the time of the survey. The statistics are weighted, but differences are not regression-adjusted 
because the assumption of balance between groups is no longer valid. 
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e Through connections with college partners, Year Up produced substantial increases in 
college enrollment during the first follow-up year. Enrollment rates for the treatment 
group fell below rates for the control group in Year 2. 


NSC records show that nearly 60 percent of treatment group members, but only 18 percent of 
control group members, were enrolled in college in the first full quarter after random 
assignment (Exhibit 6-6). By the end of Quarter 3, the treatment group members had on 
average accumulated a 2.1-month advantage in total full-time equivalent (FTE) months enrolled 
compared with control group members (Exhibit 6-7). 


Exhibit 6-6: Impact on College Enrollment in Successive Follow-Up Quarters 
Percent Enrolled (%) 
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SOURCE: Match to college records in the National Student Clearing House for 1,668 treatment and 871 control group members. 
NOTES: Statistically significant in a two-tailed test: * at the 10-percent level, ** at the 5-percent level, *** at the 1% level. 


College enrollment in the treatment group fell sharply as its members left Year Up—declining 
from 47 to 16 percent between Quarters 3 and 4 and remaining below control group rates 
thereafter. The percentages of students in the study sample with any college enrollment 
between Quarters 4 and 7 were 26 percent for the treatment group and 33 percent for the 
control group. The first three quarters of Year 3 (follow-up Quarters 8-10) saw slight increases 
in enrollment for the treatment group and slight declines for the control group. As a result, 
differences in enrollment were no longer statistically significant. 


As a legacy of large positive enrollment impacts in the first year, treatment group members 
maintained an advantage in average cumulative FTE months of enrollment through Quarter 10 
(Exhibit 6-7). 


°* ‘Treatment group members were 2-4 percentage points less likely to be enrolled in college full-time than their 


control group counterparts, however: in Quarter 10, for example, the percentages of treatment and control 
group members enrolled in college full-time were 8.1 and 11.5 percent, respectively (not shown in exhibit). 
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Exhibit 6-7: Impact on the Cumulative Number of Full-Time Equivalent Months of College 
Enrollment in Successive Follow-Up Quarters 
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SOURCE: Match to college records in the National Student Clearing House for 1,668 treatment and 871 control group members. 
NOTES: Y-axis indexes the average total number of months of full-time equivalent enrollment for each group in successive quarters. 
Statistically significant in a two-tailed test: * at the 10-percent level, ** at the 5-percent level, *** at the 1% level. 


e Year Up had no impact on receipt of college credentials in the first 18 months after 
random assignment, but it did increase the average number college credits earned. The 
program also increased receipt of industry certifications. 


As of the follow-up survey, similar small percentages (three percent) of treatment and control 
group members reported receiving credentials from a college or other education and training 
provider (Exhibit 6-8). On the other hand, more than twice as many treatment group members 
as control group members (31 percent vs. 13 percent) received exam-based certifications from 
industry and other issuing agencies. 


Whether the treatment group’s advantage in total FTE months affects college completion in the 
long term depend in part on whether these months brought an advantage in college credits 
earned. As of the 18-month survey, treatment group members reported earning on average 12 
college credits—four credits more than control group members. Future PACE reports will assess 
whether this credit advantage translates into positive impacts on college credentials in the 
longer term. 
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Exhibit 6-8: Impacts on Postsecondary Education Outcomes 
Treatment Control Impact Standard _ Relative 
Outcome Group Group (Difference) Error Impact _p-Value 

Any College Enrollment (%): 
In Quarters 0 to 3 63.7 32.5 +31.2*" (1.8) 96.0 <.001 
In Quarters 4 to 7 25.6 32.7 -7.2** (1.8) -22.0 0.035 
In Quarters 8-10 23.9 25.6 -1.7 (1.7) -6.6 0.336 
In the survey interview quarter 18.3 26.5 -8,1*** (2.0) -30.6 <.001 
Ata Year Up partner college in the 6.0 3.9 40. ee (1.0) 53.8 <001 
survey interview quarter 
Sample size 1,668 871 
Other College Outcomes 
Average number of credits received 12.2 8.2 +4,0*** (0.7) 48.8 <.001 
Percentage receiving credentials from: 

A college 3.4 3.0 +0.4 (0.8) 13.3 0.640 

Another education-training institution 8.2 7.2 +1.0 (1.3) 13.9 0.437 

A licensing/certification body 31.1 12.7 +18.5 *** (1.8) 145.7 <.001 

Any source 37.2 16.3 +20.9 ** (2.0) 128.2 <.001 
Sample size 1,301 638 


SOURCES: Statistics on college enrollment in the top panel are based on college records from the National Student Clearinghouse. 
Statistics in the bottom panel are based on the PACE 18-month follow-up survey. 

NOTES: Statistically significant in a two-tailed test: * at the 10% level; ** at the 5% level, *** at the 1% level. Statistics under 
“Relative Impact” represent impacts in column 3 as a fraction of the corresponding control group mean (i.e., 100 * [impact/control 
group mean)). 


6.3. Impacts on Other Outcomes 


Increased earnings from career-track employment in theory should generate positive impacts in 
other domains, as discussed in Chapter 3. This section summarizes findings on sample 
members’ financial status, psycho-social skills, and family formation. Because these outcomes 
are more distal in the theory of change—coming in the wake of main impact on earnings—it 
was uncertain that they would emerge in the short-term period covered by the 18-month 
survey. Hence, the analysis treats them as exploratory hypotheses and applies two-tailed tests. 


e Year Up had positive impacts on several measures of financial status linked to higher 
earnings. 


In principle, increased income from earnings should improve financial status and ease financial 
strain. Exhibit 6-9 shows that Year Up generated an 8-percentage point reduction in the 
percentages of students experiencing financial hardship, a 5-point reduction in public assistance 
receipt, and a 4-point increase in the percentage with health insurance. 


The program had at most small impacts on two measures of environmental stress—a seven- 
item index of life challenges interfering with school, work, or family responsibilities and a 
standard four-item index of perceived stress. Only the reduction in life challenges is statistically 
significant (p<.10). Effects may be small because these indicators capture relatively diffused and 
distal influences or because positive events—such as landing a challenging job—also can 
increase stress. 
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Exhibit 6-9: Impacts on Financial Status and Related Strains 
Treatment Control Impact Standard Relative p- 
Outcome Group Group (Difference) Error Impact Value 

Percent with health insurance 81.9 77.6 +4.4* (2.0) 5.7 0.026 
Percent in household receiving cash/in- 49.6 55.1 5 5u (2.2) 10.0 0.012 
kind supports 
Percent experiencing financial hardship 
the last 12 months 34.8 42.9 -8.0"* (2.3) -18.6 <.001 
Index of life challenges (1-5) L51 1.55 -0.04* (0.02) -0.09 0.069 
Index of perceived stress (1-4) 2.11 2.15 -0.04 (0.04) -0.06 0.229 


SOURCE: PACE 18-month follow-up survey. 

NOTES: Statistically significant in a two-tailed test: * at the 10% level; ** at the 5% level, *** at the 1% level. Statistics for indices (life 
challenges, perceived stress) are effect sizes (impact/standard deviation for control group), whereas statistics for other outcomes 
are relative impacts (100*[impact/control group mean]). See Appendix A for definitions of outcome measures. 


e There was little evidence of impacts on psycho-social skills or family formation. These 
results for psycho-social skills are inconclusive, however, given weaknesses in outcome 
measures. 


Results for indicators of several psycho-social skills—grit, savvy, and core self-evaluation—are 
also small and mixed.” Findings show no impact for grit, a small negative impact on savvy, and 
a small positive impact on core self-evaluation (see top panel of Exhibit 6-10). 


Readers should view these findings with caution. Recent research on application of these 
measures to intervention research has raised concern about a problem known as “reference 
bias” —a bias that can arise if interventions alter the standards respondents apply in rating 
themselves.°* The bias could result in treatment group members rating themselves as less 
skillful than control group members because the former apply a higher standard than the latter. 
Reference bias seems possible given Year Up’s generally high expectations for professional 
behavior. Standards for self-ratings on workplace savvy also might be elevated if treatment 
group members are working in professional settings with higher expectations for performance. 


2 See Appendix Exhibit D-2 for sources for these indices. A brief note on the Savvy index is warranted as it is 


likely to be less familiar to readers. ACT Inc. developed this index to measure employees’ “tendency to read 
other people’s motives, understand office politics, and anticipate the needs and intentions of others.” The 
skills required to demonstrate such tendencies are likely to be greater in professional environments with more 
complex and nuanced approaches to organization, management, and communication. 


°3 See Duckworth and Yeager (2015) and West et al. (2015). 
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Exhibit 6-10: Impacts on Psycho-Social Skills and Family Formation 
Treatment Control Impact Standard Relative p- 

Outcome Group Group __ (Difference) Error Impact Value 
Indices for Psycho-Social Skills (average) 
Grit 3,11 3.11 +0.01 0.02 +0.01 158 
Savy 4.86 4,93 -0.07* 0.03 -0.12 014 
Core self-evaluation 3.34 3.29 +0.05* 0.02 +0,11 018 
Family Structure (%) 
Living with spouse 11.9 10.6 +1.3 14 12.3 332 
Had child since random assignment / 103 12.9 27 2A 20.9 67 
currently pregnant 
Not living with parents 37.5 36.4 +11 2.0 3.0 098 


SOURCE: PACE 18-month follow-up survey. 

NOTES: Statistically significant in a two-tailed test: * at the 90% level; ** at the 95% level, *** at the 99% level. Statistics for indices 
(Grit, Savvy, Core Self-Evaluation) are effect sizes (impact/standard deviation for control group), whereas statistics for other 
outcomes are relative impacts (100*[impact/control group mean]). Childbearing/pregnancy statistics apply to female sample 
members (539 treatment and 264 control group respondents). See Appendix A for definitions of outcome measures. 


Estimated impacts on several family formation outcomes were small and not statistically 
significant (bottom panel of Exhibit 6-10). It is important to recognize that the time frame 
covered in the 18-month follow-up was very short, allowing little time to observe any career- 
related postponement of marriage and childbearing. Furthermore, the sample was relatively 
young when it entered the study. Few control group members were living with a spouse (11 
percent) at the time of the survey interview or had had a baby (13 percent) since random 
assignment. Although one might expect young adults to prefer to live on their own given 
increased earnings, nearly identical majorities of treatment and control group members (63 and 
64 percent, respectively) were still living with their parents as of the 18-month follow-up. 


6.4 Subgroup Differences in Impacts on Earnings and College Enrollment 


Knowing who benefits more and less from Year Up can help in strengthening services for 
certain types of students. Such information also can help in targeting recruitment to young 
adults most likely to benefit from the program. 


This section looks at variation in impacts on average quarterly earnings in Quarters 6-7 (the 
study’s confirmatory outcome) and on impacts on any college enrollment during Quarters 4-7 
(the second follow-up year). The analysis examines subgroup variation in impacts across nine 
characteristics, as well as across Year Up’s eight local offices. 


For each characteristic, Exhibit 6-11 gives results for two kinds of statistical significance tests. 
Daggers for each baseline characteristic summarize whether differences in impacts between 
subgroups are statistically significant. Stars indicate whether impact estimates for each 
subgroup differs from zero. 


The table shows the absolute and relative impacts for each subgroup. Absolute impacts 
represent the size of impacts in their original units and are calculated as the difference between 
average outcome levels. Relative effects express impacts in terms of the proportionate 
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increases/decreases they represent relative to the control group mean and are calculated by 
dividing impacts by control group means and multiplying by 100." 


e Subgroup analyses showed positive earnings impacts for every subgroup and local office 
examined. 


Year Up’s impact on quarterly earnings was at least $1,000 and statistically significant in every 
category examined (a relative increase of at least 30 percent over control group means). Thus, 
even at their smallest, Year Up’s impacts are among the largest that rigorous evaluations have 
identified to date (see review in Chapter 1). The size of impacts nonetheless varied across 
subgroups for a number of characteristics. 


e Starting with several basic demographic characteristics, differences in impacts by age and 
gender were not statistically significant, but impacts did vary by race-ethnicity. 


The first three panels of Exhibit 6-11 summarize subgroup results for age, gender, and race- 
ethnicity. 


Age. During site visits, several program staff conjectured that, compared with younger 
students, older students were in a better position to benefit from Year Up due to greater 
maturity and higher motivation from longer experience in unsatisfying low-wage jobs. The 
findings do not support this hypothesis: differences by age are not statistically significant, and 
point estimates show larger positive earnings impacts for younger sample members (see first 
panel of Exhibit 6-11, columns 3 and 4). 


Impacts on college enrollment do show statistically significant differences by age (see upper 
right hand panel of Exhibit 6-11). For reasons that are unclear, negative enrollment impacts 
appear for the youngest and oldest sample members, but not for those in the middle age 
group. 


Gender. Given predominance of men in the IT and financial sectors, one might expect programs 
focused on these occupations to encounter more barriers in increasing earnings for women 
than for men. Simple outcome statistics for PACE treatment group members would appear to 
align with this expectation: the percentage working in targeted occupations at $15 or above per 
hour four months after graduation was 13 percentage points lower for women than for men.” 


It is therefore noteworthy that the program’s impacts on earnings do not differ significantly for 
women and men. Apparently, women on net gained at least as much from Year Up 
participation as men, despite their relative disadvantage in the program’s target occupations. 


‘The relative impacts provide additional perspective on the importance of impacts of varying sizes, given 


differences in control group means across subgroups. For example, although an impact of $2,000 adds the 
same amount to purchasing power for a subgroup earning $10,000 as for one earning $20,000, the difference 


represents a more dramatic improvement in material circumstances for the former than for the latter. 


°° Results not shown in exhibit. 
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Exhibit 6-11: Impacts on Earnings and College Enrollment by Subgroup* 
Quarterly Earnings in Quarters 6-7 ($) College Enrollment in Quarters 4-7 (%) Sample Size 
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) 
Impact % Impact Impact § % Impact 
Characteristic Treatment Control (1)-(2) (3)/(2) x 100 Treatment Control  (5)-(6) —(7)/(6) x 100 Treatment Control 

Age ns tt 

<20 4,912 2,705 2,206 ** 81.6 26.5 40.0 -13.5* -33.8 419 219 

20-22 5,480 3,564 1,916 ** 53.8 28.0 30.0 -2.0 -6.7 782 386 

23-24 5,880 4,291 1,588 ** 37.0 20.2 29.9 -9.7 ** -32.4 437 253 
SSS Se ee ee ee 

Male 5,475 3,734 1,740 ** 46.6 23.6 29.4 5.8% -19.7 968 506 

Female 5,425 3,308 2,117 ** 64.0 28.3 37.5 -9.2 *** -24.5 670 352 

Race-Ethnicity if ns 

Hispanic, any race 6,269 4,097 2,171"* 53.0 26.8 34.8 -8.0* -23.0 540 286 

Black, non-Hispanic 4,603 3,026 15i7 52.1 23.9 28.4 -4.4* -15.5 848 441 

White/another, non-Hisp. 6,555 4,180 2,374 ** 56.8 28.4 44.4 -16.0*** -36.0 250 131 


Usual High School Grades Tt ns 


A’s and B's 5,677 3,478 2,199 *** 63.2 28.6 34.8 -6.2 ** -178 961 527 
C’s or below 5,152 3,689 1,463 ** 39.7 20.8 29.3 -8.5 *** -29.0 677 331 
Educational Attainment il ns 

High school 4,892 3,338 1,553" 46.5 19.0 28.9 -9.9 *** -34.3 838 461 
<1 year college 5,602 3,426 2,176 *** 63.5 29.2 31.4 -2.1 -6.7 369 184 
1+ year college 6,469 4,152 2,317" 55.8 36.1 42.1 -6.1 -145 431 213 
Training Commitment if ns 

Low 5,496 3,309 2,186 *** 66.1 22.9 29.1 -6.3* -21.6 549 279 
Medium 5,588 3,530 2,058 *** 58.3 28.9 32.1 -3.2 -10.0 501 283 
High 5,291 3,823 1,468 ** 38.4 25.2 36.8 -11.6 *** -31.5 588 296 

Continued 
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Quarterly Earnings in Quarters 6-7 ($) College Enrollment in Quarters 4-7 (%) Sample Size 
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) 
Impact % Impact Impact % Impact 
Characteristic Treatment Control (1)-(2) (3)/(2) x 100 Treatment Control  (5)-(6) — (7)/(6) x 100 Treatment Control 
Depressive Symptoms ns ns 

Low 5,565 3,425 2,140 ** 62.5 23.5 33.7 -10.2** -30.3 666 359 
Medium 5,627 3,993 1,635 ** 40.9 27.3 29.4 -2.1 -7.1 346 212 
High 5,210 3,407 1,803 ** 52.9 26.8 34.0 -7.2** -21.2 626 287 
Life Challenges ns ns 
Low 5,998 3,833 2,165 *** 56.5 26.2 36.6 = -10.4** -28.4 636 334 
Medium 5,442 3,651 1,792 ** 49.1 23.7 30.2 -6.4* -21.2 477 256 
High 4,797 3,131 1,666 ** 53.2 26.3 30.4 41 -13.5 525 268 
Expected Work Hours ns ns 
<10/week 5,102 2,998 2,104 ** 70.2 27.8 38.2 -10.4 *** -27.2 643 348 
10-29/week 5,889 4,010 1,878 ** 46.8 25.2 30.9 -5.7* -18.4 719 359 
30+/week 5,205 3,780 1,425 ** 37.7 22.5 25.6 -3.0 -11.7 276 151 
Office iit ttt 
A 4,192 2,825 1,367 ** 48.4 29.5 36.8 -7.2 -19.6 161 85 
B 5,994 4,602 1,392 ** 30.2 18.5 26.5 -8.0* -30.2 316 161 
C 6,350 4,617 1,732 ** 37.5 46.6 35.3 11.3* 32.0 156 85 
D 4,951 3,070 1,880** 61.3 19.7 17.9 18 10.1 242 120 
E 4,427 2,483 1,944 ** 78.3 22.7 34.3  -11.6* -33.8 173 98 
F 5,116 3,042 2,075 ** 68.2 25.3 33.8 -8.4* -24.9 273 149 
G 5,900 3,455 2,445 *** 70.8 21.4 39.0  -17.6*** -45.1 158 82 
H 8,096 4,430 3,666 ** 82.8 30.6 50.0 -19.4** -38.8 159 78 


SOURCES: Quarterly earnings are based on wage records obtained in a match to the National Directory of New Hires. College enrollment is based on college records from a match to 
the National Student Clearinghouse. Measures of baseline characteristics are from the PACE baseline surveys. 

NOTES: *Daggers appearing in the top row for each characteristic identify characteristics for which subgroup impacts differ t at the 90% level, tt at the 95% level, ttt at the 99% 
level, based on a two-tailed test. Stars summarize tests of whether impacts for individual subgroups are statistically significant: * Impact for subgroup differs from zero in a two-tailed test 
at the 90% level, ** at the 95% level, *** at the 99% level. See Appendix A for definitions of characteristics. 

Total sample size for treatment and control groups are 1,638 and 858, respectively, for earnings and 1,668 and 871 for college enrollment. Columns 9 and 10 show subgroup samples 
for earnings analyses. 
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Race-ethnicity. Reflecting Year Up’s longstanding commitment to inner city youth, more than 
half of its participants identify as non-Hispanic black. The special challenges facing this 
population are evident in substantially lower quarterly earnings and college enrollment for 
control group members who identify as non-Hispanic black ($3,026 and 28 percent, 
respectively) versus those who identify as Hispanic ($4,097 and 35 percent) or non-Hispanic 
white/another race-ethnicity ($4,180 and 44 percent). 


Year Up’s impact on quarterly earnings also was lower for black sample members ($1,577) than 
for the Hispanic ($2,171) and white/another race-ethnicity ($2,374) subgroups. These 
differences are statistically significant (at p<.10). The results indicate that Year Up does not 
benefit young adults who identify as non-Hispanic black to the same degree as it benefits other 
groups. 


An important caveat is that the earnings impact for blacks was still positive, statistically 
significant, and large. Indeed, the relative impact (the proportionate increase in earnings over 
the control group mean) for blacks (52 percent) is very similar to the relative impacts for the 
other two subgroups (53 and 57 percent for those identifying as Hispanic and white/another 
race-ethnicity, respectively). 


Race-ethnicity differences in college enrollment impacts were not statistically significant. Point 
estimates show the smallest reduction for blacks (4 percentage points) and the largest 
reduction for the white/another race-ethnicity group (16 percentage points). Absent Year Up, 
control group means again show less favorable outcomes for blacks (28 percent enrolling in 
college) than for the white/another group (44 percent enrolling). 


e Impacts are more favorable for young adults with stronger education backgrounds. 


High school grades reflect academic ability as well as other factors influencing school 
performance, such as self-discipline and home environment. Self-reports on high school grades 
strongly predict college enrollment and persistence (Fein 2016b). 


Usual high school grades. Among members of the control group, young adults who reported 
receiving mostly A’s and B’s in high school earned slightly less ($3,478) than those reporting 
mostly C’s or below ($3,689), possibly because the former were more likely to be in college 
than were the latter (35 and 29 percent, respectively). 


But Year Up’s impact on earnings was substantially larger for better high school performers 
($2,199) than for poorer performers ($1,463)—a difference significant at the .05 level. This 
greater earnings boost did not appear to exact a larger price in college enrollment. Reductions 
in college enrollment are similar for those earning A’s and B’s in high school (6 percent) versus 
those earning C’s and below (9 percent). 


Educational attainment. Prior college experience also may signal qualities favorable to success 
in employment and training. Supporting this notion, control group members with at least a 
year’s worth of college credit at the outset had higher earnings ($4,152) and college enrollment 
(42 percent) than control group members with less than a year of college ($3,426 and 31 
percent) or only a high school credential ($3,338 and 29 percent). 
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Differences in Year Up’s impacts on earnings by prior educational attainment also were 
statistically significant (at p<.10). Sample members with a year or more of college before Year 
Up gained $2,317, compared to $1,553 for those with only a high school credential. 


Smaller positive earnings impacts for sample members with only a high school credential were 
accompanied by a statistically significant negative impact on college enrollment (-10 
percentage points), while enrollment impacts for those with some college were smaller and not 
Statistically significant. The differences in enrollment impacts across these subgroups are not 
Statistically significant, however. 


e Earnings impacts for sample members who were most motivated at baseline were smaller 
than impacts for less motivated sample members. 


Though Year Up admissions teams aim to recruit applicants with sufficient motivation and 
supportable life challenges, the standards are flexible and result in a reasonably diverse 
clientele. As a result, the research sample provides the basis for informative comparisons 
between enrollees at different levels on these attributes. 


Training commitment. To measure motivation, the baseline survey included a 12-item index of 
training commitment developed by the testing firm ACT Inc. Values for control group members 
in Exhibit 6-11 provide some evidence for the measure’s validity: mean quarterly earnings and 
college enrollment both increase with increasing baseline training commitment.”° 


In screening, Year Up staff aim to select relatively motivated applicants, reflecting the 
assumption that motivation is important for success in the program. However, subgroup 
analyses show somewhat lower impacts for the most highly motivated sample members 
compared to less motivated ones—a statistically significant difference (at p<.10). The impact is 
$1,468 for sample members ranking in the upper third on training commitment versus $2,058 
and $2,186 for those in the middle and bottom thirds, respectively. 


One possible explanation—that Year Up leads youth with higher motivation for training to opt 
for college over work—is not supported by results for college enrollment (see right-hand panel 
of Exhibit 6-11). To the contrary, point estimates are negative and relatively large for the most 
motivated sample members (12 percentage points) compared with less motivated sample 
members (though the differences are not statistically significant). Given improved job 
prospects, young adults who are initially the most training-oriented may be especially willing to 
postpone college plans. 


Depressive symptoms/Life challenges. Impacts do not appear to be related to baseline levels 
of depression and other life challenges: differences in earnings and college enrollment impacts 
are not statistically significant for these characteristics. The result suggests that Year Up 
admitted students whose difficulties on these factors were manageable. That said, the 
depressive symptoms index is based on self-reports rather than a psychological assessment, 


°° The division into lower, middle, and upper “thirds” for Exhibit 6-11 was necessarily approximate because 


clustering created many ties on values straddling thirds. 
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and the index of life challenges developed for PACE did not include many of the risks that Year 
Up seeks to identify in screening and may not fully capture challenges constraining impact.” 


e Earnings impacts were somewhat lower for young adults who initially expected to work 
full-time while attending Year Up. 


Initial expectations for work during Year Up could affect post-program earnings impacts in 
several ways. On the one hand, earnings might be necessary to meet living expenses while 
participating in Year Up and support success in the program. On the other, working may siphon 
valuable time and energy from effort needed in the program. Another possibility is that for 
some respondents, expectations for little/no work signal additional barriers to employment 
that Year Up might or might not be able to help address. 


Although a global test showed no difference in earnings impacts across all three levels of 
expected work hours, a pairwise test shows that earnings impacts do differ significantly 
between the lowest and highest levels of expected work hours. The $679 difference between 
impacts for sample members expecting to work fewer than 10 hours a week ($2,104) and those 
expecting to work 30 or more hours a week ($1,425) is statistically significant (at p<.10). 


Bolstering the “competition for time and energy” hypothesis, separate analyses show that 
treatment group members who at baseline said they expected to work at least 20 hours/week 
in the next few months were four to six percentage points less likely to complete Year Up than 
those who did not expect to work (not shown in exhibit). 


Impacts on college enrollment also did not vary by initial expected work hours in a global test 
across the three categories. Point estimates here show statistically significant negative 
enrollment impacts only for students who initially expected to work less than 30 hours/week, 
perhaps suggesting increased work effort in this group. 


e Impacts on earnings were substantial and statistically significant for all eight local offices, 
though the size of effects varied considerably. 


Findings in Chapter 4 showed that though all eight local offices implemented Year Up with 
reasonable fidelity, some offices encountered more difficulties and scored lower on 
performance benchmarks (e.g., completion) than others. Impacts might vary across offices for 
this reason, or because the composition of program participants and other local conditions 
varied. 


Variability in the size of earnings impacts is substantial and statistically significant (p<.05). 
Estimates range from $1,367 to $3,666 (in Offices A and H, respectively), and proportionate 


°7 The index summarized the extent to which applicants indicated that difficulties with each of the following had 


interfered with school, work, or family responsibilities in the past year: child care, transportation, drug/alcohol 
use, illness/health problem, family arguments, and domestic violence. Analyses using Year Up’s own risk index 
also would have been interesting, but the program was unable to provide risk data for control group 
members. 
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impacts vary from 30 to 83 percent (Offices B and H).°° Taken together, the results suggest that 
the program was effective in a wide range of urban environments. 


Differences in college enrollment impacts also are statistically significant (p<.01) and 
considerable—varying from a 19-percentage point decline in enrollment (Office H) to an 11- 
percentage point increase (Office C). Differences in Year Up’s connections with college partners 
during the program year, discussed in Chapter 4, are likely a contributing factor. Further 
analysis indicates that the two offices with no enrollment declines in Quarters 4-7 (Offices C 
and D) were the ones generating the largest enrollment impacts in Quarters 0-3 (60 and 

68 percentage points, respectively), suggesting that more robust efforts to engage participants 
in college during the program foster longer lasting attachments to college.*° 


The offices with large reductions in college enrollment tended to have large positive earnings 
impacts. For example, the two offices with the largest reductions in college—Offices G and H, 
with 18 and 19-point declines, respectively—also had the largest earnings gains ($2,445 and 
$3,666, respectively). This finding suggests that increased time in career-track jobs after Year 
Up may reduce the time young adults are willing to devote to school, at least in the short-term. 


sid Impacts and means for Year Up’s Seattle office in this panel are adjusted to correct for under-reporting of 


wage records to NDNH by Washington State (see Section 6.1). 


°° Results not shown in exhibit. 
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7. Conclusions 


This report has provided rigorous evidence on the implementation and early impacts of the 
Year Up program. The findings show that Year Up implemented the program with high fidelity 
to its design. Earnings impacts, estimated over a three-year follow-up period after random 
assignment, are the largest found to date for workforce programs tested with an RCT design.©° 


This concluding chapter summarizes the report’s major findings (Section 7.1) and reflects on 
their wider implications (Section 7.2). A concluding section (7.3) discusses questions for future 
research in the ongoing PACE evaluation and beyond. 


7.1 Summary Assessment 


Results from field research and quantitative analyses of program data show that the eight local 
Year Up offices fully implemented most components of the program. This is a substantial 
achievement, given the number and complexity of program elements, varying local 
environments, and need for coordination across many services. 


Quantitative measures of implementation. Statistics generally attest to a high level of 
implementation. All offices met the goal of expanding recruitment by 50 percent for PACE. They 
did so while maintaining applicant quality, admitting one in six applicants as in the past. 


Participation and retention in Year Up were high: 96 percent of treatment group members 
enrolled in the program, and 81 percent (85 percent of enrollees) completed the six-month L&D 
phase. Nearly all enrollees (96 percent) received citations for at least one behavior contract 
infraction, and 45 percent received 10 or more infractions. Year Up placed virtually all Learning 
and Development phase (L&D) completers in internships, and 75 percent of the all treatment 
group members (78 percent of enrollees) finished the program. 


Four months after graduation, 83 percent of graduates were working either full-time (74 
percent) or part-time (nine percent). Of those working, 88 percent were in an occupation 
relevant to Year Up’s training, 77 percent were earning at least $15/hour, and 41 percent had a 
job with their internship sponsor. 


Year Up secured the full $920 weekly payment from employers for 96 percent of positions, 
netting an average $22,404 per placement. Revenue from employers financed 59 percent of the 
program’s $28,290 per participant cost, with most of the balance provided by foundations and 
private donors. Year Up met its goal of relying minimally on government financing: only two 
percent of overall revenue was from public agencies. 


Although generally high, outcomes varied somewhat across offices. For example, completion 
rates ranged from 64 percent in one office to 85 percent in two offices, with the remainder in 
the 75-79 percent range. Rates of co-enrollment at local college partners ranged from 12 
percent to 96 percent. Although all offices recorded at least one infraction for a very high 
fraction of enrollees, citations for 10 or more fractions ranged from 21 percent to 60 percent. 
Possible sources of variation in these outcomes include differences in local environments and 


° See literature review in Chapter 1, Section 1.3.1. 
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participant characteristics, as well as in implementation. The number of offices was too small to 
support a statistical investigation of the factors responsible. 


Culture. Year Up’s leaders promoted active, ongoing discussion of mission and values and how 
to align services with them. This emphasis on culture helped to integrate Year Up’s wide array 
of services across varied stakeholders (e.g., staff, students, college and corporate partners) and 
locations. 


In fieldwork, the PACE team documented widespread awareness of Year Up’s values and many 
examples of staff and students applying them to their activities. A notable example is the core 
value “strive to learn.” Intentional efforts to apply this value led students to take responsibility 
for their studies, local staff to put energy into improving services, national staff to create 
program-wide mechanisms for continuous improvement, and all to give and accept 
constructive feedback. 


Year Up’s culture also fosters a strong business ethos. This ethos encourages initiative and 
entrepreneurial behavior, as well as substantial use of language and tools from the business 
world. Interviews with students, staff, and employers suggested that this aspect of the culture 
helps to motivate students and staff to work hard and to connect well with Year Up’s employer 
partners. 


Leaders used the program’s performance measurement system—FM-RADIO—to promote 
alignment on common goals across and within offices. This tool provided a way to monitor and 
troubleshoot varying approaches to services while encouraging innovative adaptation to local 
conditions. Its measures of multiple outcomes helped to promote awareness of the 
relationships between work performed by different staff specialties—highlighting, for example, 
the connections between admissions screening, L&D retention, and internship revenue. 


Service receipt. Nearly all young adults assigned to the treatment group (96 percent) enrolled 
in Year Up and received at least some training. More than half (57 percent) of control group 
members also got some training, mostly at community colleges. But treatment group members 
were far more likely to cite promising instructional approaches in the courses they took (e.g., 
discussion, group projects, active learning, real-world relevance); to take courses in life skills; 
get related work experience; and to receive an array of support services (e.g., academic 
guidance and career counseling, tutoring, financial assistance, supportive services, and job 
search and placement assistance). 


Overall impacts. Year Up increased average quarterly earnings by $1,895 (53 percent) in the 
sixth and seventh quarters after random assignment—the outcome pre-specified in the main 
confirmatory hypothesis for this analysis. Large positive impacts persisted through the third 
follow-up year. Overall impacts were $5,181 (39 percent) and $7,011 (40 percent) in the second 
and third years, respectively. 


By comparison, the largest impacts from RCTs in the workforce literature have been smaller, 
particularly for young adult subsamples. As reviewed in Chapter 1, the SEIS evaluation reported 
some of the largest annual earnings impacts prior to this study—$4,001 (29 percent) in the 
second follow-up year for a adults of all ages pooled across three leading sectoral programs 
(Maguire et al. 2010). The pooled impact for young adults (18-24) was $3,092 (25 percent). 
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Earnings impacts for Project Quest, another sectoral training program, reached $5,080 (22 
percent) in the fifth follow-up year among adults of all ages, but subgroup analysis found no 
positive effects for young adults (Elliot & Roder 2017). 


Training programs typically produce negative earnings impacts in the first follow-up year, as 
participants prioritize training over work. Year Up was no exception, with a $5,338 reduction in 
earnings in the first year. The total earnings gain of $12,192 over the next two years ($5,181 + 
$7,011) more than offset this reduction. Moreover, the Year 1 earnings impacts do not count 
$7,172 in Year Up stipends received by the average treatment group member. 


Although Year Up produced strong positive earnings impacts, its impacts on college enrollment 
were negative in the second follow-up year. Treatment group members maintained a small 
advantage in cumulative months of FTE enrollment and college credits, thanks to co-enrollment 
at Year Up’s local college partners during the program. But this advantage was not sufficient to 
increase receipt of college credentials in the short-run. 


Subgroup impacts. Year Up substantially increased earnings for every subgroup of students 
examined across 11 different characteristics and across Year Up’s eight local offices. Impacts on 
quarterly earnings in Quarters 6-7 were at least $1,000 and statistically significant in every 
subgroup examined. 


Though generally large, the size of impacts varied substantially. For example, earnings impacts 
were lower for students with weaker high school performance and for those expecting to work 
full-time—suggesting possible benefits from strengthening help with academics and with 
balancing school and work (perhaps by supporting reduced work hours). Smaller earnings gains 
for blacks signal that well-documented disadvantages facing this group—including greater 
discrimination and social and economic challenges—also require greater efforts to overcome. 


Impacts were more favorable for students who rated /ower on training commitment than for 
those who rated higher at the outset. This finding raises the possibility that Year Up may be 
effective for less motivated young adults than the program historically has targeted. Perhaps 
low motivation is malleable for some young adults, given sufficient time in a highly structured 
and supportive program. 


Substantial positive impacts in all eight local offices attest to Year Up’s replicability in different 
urban environments. The size of impacts nonetheless varied substantially across offices. This 
variation might arise from differences in local environments, the quality of implementation, or 
both. Considerable differences in average control group earnings across offices indicate that 
contextual factors—such as local population characteristics, labor markets, and access to 
transportation—could play an important role. But with only eight offices and many dimensions 
to local context and implementation effectiveness, it is not possible to identify contributing 
factors with any confidence. 


Continuous improvement. Though mature in many respects, Year Up continued to evolve 
during and after the PACE years (2013-2014). National program improvement efforts sought to 
address challenges arising from rapid growth and create a stronger platform for future 
upscaling. These challenges included some divergence from desired practices and some over- 
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compartmentalization of specialized staff functions. In response, national staff took a series of 
steps to strengthen monitoring and improve service coordination. 


Leaders’ response also included steps to standardize services and shift authority to the national 
level. The 2013-2014 period brought initiatives to standardize admissions processes, centralize 
curriculum development, expand flows of interns to large employers, and extend the period of 
post-graduation employment supports. 


Rapid growth and high performance expectations increased pressures on staff, and a number of 
interviewees described the work environment as involving high levels of stress and staff 
turnover. In response, Year Up’s leaders advised national staff to minimize unnecessary 
demands on local staff, introduced wellness initiatives, and sought to hire staff highly 
committed to the organization’s mission. 


Recognizing limits to upscaling a model requiring ongoing infusions of philanthropic funding, in 
2014, Year Up decided to stabilize the size of its stand-alone core program (the version 
evaluated in PACE) and concentrate efforts to scale on adaptations for college and workplace 
settings. The revised models aim to reduce costs by leveraging instruction and other resources 
at partner institutions.°* 


7.2 Implications 


This report’s findings add to growing evidence on the promise of well-implemented career 
pathways approaches with a strong sectoral focus. These approaches tend to be intensive and 
comprehensive, address psycho-social skills as well as occupation-specific competencies, 
provide opportunities for work-based learning, engage employers, incorporate evidence-based 
practices, and emphasize continuous improvement. 


The PACE findings should allay concerns that programs like Year Up that screen for ability, 
motivation, and manageable life challenges necessarily are serving young adults who would do 
just as well without such interventions. Baseline statistics show substantial socioeconomic 
disadvantages in the young adults Year Up recruited, and the study’s random assignment 
design—providing a valid comparison to similar non-recipients—establishes that they truly 
benefitted from the program’s services. 


Stringent screening does raise the question of whether Year Up is appropriate for more than a 
narrow segment of Opportunity Youth. This study’s findings of large positive impacts for 
participants with weaker academic backgrounds, lower motivation, lower incomes, and a wide 
variety of demographic characteristics hint that the program might be effective with lower- 
skilled, more disadvantaged populations. But they do not prove that it would be more widely 
effective or establish how far beyond the current population wider benefits might extend. 
Future experiments testing the model with expanded target populations would be valuable. 


Year Up’s success in engaging employers demonstrates potential for not only expanding 
opportunities for work-based learning in fast-growing professional occupations, but also for 
mobilizing private-sector financing of organizations that serve as intermediaries between newly 


®* See Fein (20126a) for a description of Year Up’s approach to scaling up. 
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skilled job seekers and employers. Thus, while exemplifying more familiar forms of engagement 
such as appointing business representatives to advisory boards and working together in 
identifying skills in demand and designing training programs, Year Up’s experience also 
highlights the potential for employers to play central roles in funding and service delivery in 
workforce programs for economically disadvantaged adults. 


Several employers noted that experience with Year Up interns led them to revamp career 
pathways in technical occupations to create new career entry points at lower skill levels. One 
large firm re-oriented its entry-level hiring practices to recruit sub-baccalaureate-level interns. 
For another, creating assignments for interns stimulated the realization that a layer of tasks 
currently handled by mid-level employees could be performed by Year Up graduates. Although 
anecdotal, such reports support the notion that effective sectoral programs can lead employers 
to substitute a trusted workforce intermediary’s “brand” for academic credentials in hiring 
decisions. 


Year Up occupies an important position in the continuum of demand-focused workforce 
interventions—a continuum ranging from programs offering several weeks of training to multi- 
year career pathways initiatives and apprenticeship programs. That Year Up is more effective 
than the former bolsters the case for more intensive interventions with substantial training, 
supports, and employment experience. Compared with traditional apprenticeships, Year Up 
provides a shorter path to good-paying jobs, and its internships require less up-front 
commitment from employers. This greater flexibility and freedom may fit hiring preferences at 
many U.S. firms better than traditional European-style apprenticeships (Hoffman 2011).°* 


Another concern sometimes raised is that the effectiveness of sectoral approaches may be 
limited to times when targeted jobs are plentiful—or to localities where relevant industries are 
thriving. This report’s positive findings apply to a period of accelerating recovery from 
recession. But a small preceding experiment also found substantial positive impacts for Year Up 
at the height of the Great Recession (Elliot and Roder 2014). Thus, there is some reason to 
believe that programs such as Year Up can be effective in different economic climates. 


Year Up has recognized that its services to date have operated on a small scale relative to 
projected shortfalls in skilled technical workers and the number of Opportunity Youth who 
could benefit from services. In the organization’s estimation, substantial increases in scale will 
require addressing the major workforce needs of large firms. Employers interviewed for PACE 
cited a mix of motives for hosting Year Up interns—including contributions to society as well as 
screening for prospective hires. Year Up leaders believe that large-scale expansion will require 
assessing and addressing the larger workforce challenges facing employers. The organization is 
in the process of re-engineering its approach to test this proposition. 


7.3. Remaining Questions 


While answering one set of questions about Year Up’s implementation and early impacts, this 
report also raises many questions for future research. Future data collection and analysis for 


* Consistent with such preferences, recent U.S. apprenticeship initiatives encourage shorter-term training 


programs in IT and other professional fields. See https://www.dol.gov/apprenticeship/index.htm. 
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the PACE evaluation will address some of these questions, whereas others will require separate 
studies. 


One series of questions for the ongoing PACE evaluation concerns how Year Up’s impacts will 
evolve over the longer-term: 


e Do impacts on earnings persist or fade? 


e To what extent do the program’s positive initial impacts on employment in entry-level 
career-track jobs position young adults to continue along career pathways in 
occupations Year Up targets? 


e What role does college play in participants’ career pathways, compared to young adults 
who did not go to Year Up? At what rates do Year Up graduates return to school and 
earn college credentials in the longer-term? 


e To what degree does career progress affect well-being in other domains? 


Future reports on program impacts covering up to six years of follow-up will address these and 
other questions. 


Another important set of questions concerns Year Up’s costs and benefits. A planned cost- 
benefit study will estimate the net financial returns from the perspectives of participants, 
government, the balance of society, and society as a whole (i.e., summing the first three). 
Positive earnings impacts and unique features of Year Up’s financial model raise interesting 
questions about longer-term costs and benefits. Although Year Up’s costs per participant are 
relatively high, revenue from employers covers a substantial share of costs, and dependence on 
public funds is minimal. From participants’ perspective, foregone earnings during the program 
year are more than offset by program stipends. The potential societal benefits are considerable: 
one set of simulations estimates that the average lifetime cost of being disconnected from 
education and work during young adulthood is $600,000 per youth (Belfield et al. 2012).° 


Year Up’s model also raises important questions about returns on employers’ investments in 
interns: Do benefits from reduced hiring and turnover costs, increased productivity, tax 
deductions for charitable giving (for those booking their investments in this way), and improved 
public perceptions of companies exceed the costs of what they pay Year Up for interns? Analysis 
of these questions falls outside the PACE evaluation’s scope and would need to be pursued as a 
separate study. 


As discussed in Chapter 3, the PACE evaluation design generates rigorous estimates of Year 
Up’s overall effects, but does not support rigorous assessment of the contributions of its 


The figure is Belfield et al.’s (2012) estimate for the roughly half of all Opportunity Youth who are “under- 


connected” —defined as those with some spells of school or work while aged 16-24. This segment of the 
population is more similar to Year Up’s target population than the remaining half they describe as “chronically 
disconnected” youth (i.e., those with virtually no school or work from 16-24). Their calculations represent 
lifetime estimates for the total cost to society in lost earnings and tax revenue, increased costs of crime, 
diminished health and higher health care costs, and higher welfare payments minus “savings” from diminished 
use of public education subsidies. The figure does not include potential benefits from improved economic 
growth that a more adequate supply of skilled technical workers might bring. 
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components. As outlined below, some non-experimental analysis could be fruitful within the 
PACE design, but rigorous estimates will require new experiments. 


Within the current design, one approach to identifying influential ingredients would be to trace 
the presence and absence of effects on intermediate outcomes specifically targeted by program 
components or clusters of components. For example, one set of Year Up components aims to 
improve earnings prospects by strengthening skills—including the general cognitive and 
psycho-social skills many employers cite as critical needs, as well as technical skills (Accenture 
et al. 2014). A separate set of services involve recruiting and matching suitable young adults 
with positions employers need to fill. The two sets of components—training and recruitment- 
matching—are intended to be synergistic, but may contribute differentially to overall impacts. 


The relative effects of skills training could be analyzed by measuring the targeted skills and 
statistically estimating the degree to which impacts on skills explain (or “mediate”) impacts on 
earnings. To date, PACE has attempted to measure just a few of the many skills that Year Up 
targets, and, as noted in Chapter 6, recent research raises serious questions about the validity 
of self-report items like those in the PACE follow-up survey. So supplemental data collection— 
involving innovative survey methods and direct skills assessment approaches (e.g., performance 
testing) for the Year Up sample—would be needed. 


More rigorous estimates of the incremental contributions of program components could be 
obtained through randomized experiments selectively deleting or adding components of 
interest to the base model. Such experiments would need to be limited to components that 
could be varied. 


Experiments also could be useful in testing revised approaches to selected services. Year Up 
already has one such experiment underway. The test focuses on the effects of strengthened 
approaches to monitoring and supports for students who are struggling academically. The study 
is part of a parallel evaluation of Year Up’s newer Professional Training Corps (PTC) model (an 
adaptation of the core program for college settings). The PTC study also includes an RCT 
measuring the program’s overall impacts. 


A third promising category of experiments is tests of Year Up’s effectiveness for wider 
populations. Such experiments might test expansions to youth with lower skills and more life 
challenges than the program currently targets, to younger and older age groups, and to youth 
in non-urban settings. 


Finally, replication has been the Achilles’ heel of many promising workforce programs. Although 
Year Up has been effective in expanding its own operations, it is unclear whether and how 
other organizations might be equipped to replicate Year Up’s approach. In particular, Year Up’s 
capacity to connect with the business world took a highly entrepreneurial team of former 
business executives over a decade to develop. Whether and how this capacity can be 
transferred to a wider array of workforce intermediaries, postsecondary education providers, 
and employers are among the more important questions raised by Year Up’s success. 


For more on this work, see http://www.abtassociates.com/Perspectives/November-2016/Learning-Together- 


Building-Stronger-Practitioner-R. 
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